monitoring at cloud scale
DESCRIPTION
Conference given on June 13th 2013 in AmsterdamTRANSCRIPT
;
Monitoring at Cloud ScaleMonitoring at Cloud Scale
Julien PivottoJulien Pivotto
Build a cloud Day AmsterdamBuild a cloud Day AmsterdamJune 13th, 2013June 13th, 2013
;
IntroductionAround monitoring
ToolsConclusion
Table of contents
1 IntroductionDevOpsmonitoringsucksmonitoringsucks
2 Around monitoringThe cloudThe pastEnvironmentChallengesInfrastructure as code
3 ToolsCollectdLogstashStatsdGraphiteIcinga
4 ConclusionThey work togetherSharingThere are solutions
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
Julien PivottoJulien Pivotto• sysadmin @ inuitssysadmin @ inuits• open-source defender for 7+ yearsopen-source defender for 7+ years• devops believerdevops believer• @roidelapluie on twitter/github@roidelapluie on twitter/github
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
DevOpsmonitoringsucksmonitoringsucks
What is that DevOps stuff again?
• CultureCulture• (Lean)(Lean)• AutomationAutomation• MeasurementMeasurement• SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
DevOpsmonitoringsucksmonitoringsucks
#monitoringsucks
• https://github.com/monitoringsuckshttps://github.com/monitoringsucks• a movement to find a solution to monitoringa movement to find a solution to monitoring• the feeling that monitoring is stucked in the pastthe feeling that monitoring is stucked in the past
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
DevOpsmonitoringsucksmonitoringsucks
#monitoringlove
• then it turned into #monitoringlovethen it turned into #monitoringlove• relevant tools existrelevant tools exist• they just need to be usedthey just need to be used• following the unix philosophyfollowing the unix philosophy
we are going to explore some of themwe are going to explore some of them
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
What is different in the cloud?
• ScaleScale• VelocityVelocity• More changes, more oftenMore changes, more often
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
What do you need?
• scalabilityscalability• automationautomation
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
time for retirement
• forget all-in-one toolsforget all-in-one tools• forget auto-discovery toolsforget auto-discovery tools• forget non-scalable toolsforget non-scalable tools• forget tools you can not automateforget tools you can not automate
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
ZabbixZabbixJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
CentreonCentreonJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
GroundWorkGroundWorkJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
CactiCactiJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
HypericHypericJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
BigBrotherBigBrotherJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
MuninMuninJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
ZenossZenossJulien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Your infrastructure today
http://www.flickr.com/photos/bjbrake/235217140/
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Your infrastructure tomorrow
http://www.flickr.com/photos/bjbrake/235217140/
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Your infrastructure in 6 months
http://www.flickr.com/photos/bjbrake/235217140/
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Challenges
• ReproducabilityReproducability• SpeedSpeed• MetricsMetrics• OrchestrationOrchestration
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
WANTED
http://www.flickr.com/photos/pagedooley/3124443099/
• Small toolsSmall tools• Collect / MangleCollect / Mangle• Analyse / ActAnalyse / Act• VisualizeVisualize
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
WANTED
The UNIX philosophyThe UNIX philosophy
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Automation
• One source of trust: puppet, chef, . . .One source of trust: puppet, chef, . . .• Exported resourceExported resource• Monitor in the same location you deployMonitor in the same location you deploy• Infrastructure-as-CodeInfrastructure-as-Code• no autodiscovery toolsno autodiscovery tools
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Automation
If it is not automated || not monitoredIf it is not automated || not monitoredthen it does not exist!then it does not exist!
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
The cloudThe pastEnvironmentChallengesInfrastructure as code
Example in puppet
• Create a definition for your applicationCreate a definition for your application• In that definition, add the configuration, the vhosts. . .In that definition, add the configuration, the vhosts. . .• Then export the monitoring (@@icinga_service)Then export the monitoring (@@icinga_service)• In bonus you can export DB configuration, etc. . .In bonus you can export DB configuration, etc. . .• Use only the "meta" definitionUse only the "meta" definition• Collect the exported ressources (Nagios_service <||>)Collect the exported ressources (Nagios_service <||>)
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Collectd
• Statistics collection daemonStatistics collection daemon• A lot of plugins available. . .A lot of plugins available. . .• Can send data to graphiteCan send data to graphite• Simple configurationSimple configuration
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Collectd plugins
http://www.flickr.com/photos/juhansonin/3141561416/
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Collectd plugins
AMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Loadgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt LoadLogFile LPAR MadWifi MBMon memcachec memcached Memory ModbusLogFile LPAR MadWifi MBMon memcachec memcached Memory Modbus
Monitorus Multimeter MySQL NetApp Netlink Network NFS nginxMonitorus Multimeter MySQL NetApp Netlink Network NFS nginxNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSockTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSock
Uptime Users UUID Varnish vmem VServer Wireless XMMSUptime Users UUID Varnish vmem VServer Wireless XMMSWrite_Graphite Write_HTTP Write_MongoDBWrite_Graphite Write_HTTP Write_MongoDB
Write_Redis Write_Riemann ZFS_ARCWrite_Redis Write_Riemann ZFS_ARC
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Logstash
• Ship logs from any sourceShip logs from any source• Filter themFilter them• Index themIndex them• Search themSearch them• Backed with elasticsearchBacked with elasticsearch
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Kibana
http://kibana.org/images/screenshots/searchss.png
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Statsd
• Stats aggregationStats aggregation• Simple countersSimple counters• Flushes every XX seconds to graphiteFlushes every XX seconds to graphite• Text over UDPText over UDP
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Statsd
echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Graphite
• Graphing made simpleGraphing made simple• A lot of helpers functionsA lot of helpers functions• Listening on UDP and TCPListening on UDP and TCP• Text over UDP/TCPText over UDP/TCP
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Send data to graphite
echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Graphite API
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
gdash
https://github.com/ripienaar/gdash
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Icinga
• Fork of nagiosFork of nagios• Large and vibrant communityLarge and vibrant community• Configuration compatible with nagiosConfiguration compatible with nagios• User-friendly interfaceUser-friendly interface• Use Icinga Classic!Use Icinga Classic!
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
CollectdLogstashStatsdGraphiteIcinga
Icinga
https://icinga.org
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
Toolchain from apache to nagios
• Apache ships logs to rsyslogApache ships logs to rsyslog• Rsyslog ships logs to logstashRsyslog ships logs to logstash• Logstash ships metrics to statsdLogstash ships metrics to statsd• Statsd ships metrics to GraphiteStatsd ships metrics to Graphite• Icinga query metric from graphiteIcinga query metric from graphite• https://github.com/etsy/nagios_toolshttps://github.com/etsy/nagios_tools
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
Reusing Icinga/Nagios perfdata
• Icinga performs various checksIcinga performs various checks• Icinga sends perfdata to graphiteIcinga sends perfdata to graphite• Graphite stores the dataGraphite stores the data• Gdash serves them inside dashboardsGdash serves them inside dashboards• https://github.com/roidelapluie/icinga-to-graphitehttps://github.com/roidelapluie/icinga-to-graphite
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
The metrics
• Everything can become a metricEverything can become a metric• Performance metricsPerformance metrics• Usage metricsUsage metrics• Business-valuable metricsBusiness-valuable metrics• People metricsPeople metrics• Metrics are knowledgeMetrics are knowledge
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
Metrics that matter
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
What have we seen?
• We have seen only open-source softwareWe have seen only open-source software• Small, pluggable daemonsSmall, pluggable daemons• Robust solutionsRobust solutions• Nice & user-friendly outputNice & user-friendly output• They play togetherThey play together
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
Homework
• SensuSensu• RiemannRiemann• dashingdashing• ExtreMonExtreMon• EsperEsper• SkylineSkyline• OculusOculus
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
Try them yourself
https://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-puppet-logstashhttps://github.com/KrisBuytaert/vagrant-puppet-logstash
Julien Pivotto Monitoring at Cloud Scale
;
IntroductionAround monitoring
ToolsConclusion
They work togetherSharingThere are solutions
Contact
Julien PivottoJulien [email protected]@inuits.eu@roidelapluie@roidelapluie
INUITS bvbaINUITS bvbaDuboisstraat 50Duboisstraat 502060 Antwerp2060 AntwerpBelgiumBelgium+32 473 441 636+32 473 441 636https://inuits.euhttps://inuits.eu
Julien Pivotto Monitoring at Cloud Scale