osdc 2015: tudor golubenco | application performance management with packetbeat, elasticsearch and...
TRANSCRIPT
![Page 1: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/1.jpg)
Application performance management with PacketBeat, Elasticsearch and Kibana
Tudor Golubenco (@tudor_g)
![Page 2: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/2.jpg)
What is PacketBeat? ¯\_(ツ)_/¯
![Page 3: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/3.jpg)
What is PacketBeat
• “Open Source Application Monitoring”
• “Monitoring & Troubleshooting for Distributed Applications”
• “Distributed Wireshark with a lot more analytics features”
• “Application Performance Management”
![Page 4: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/4.jpg)
How it works? ಠ_ಠ
![Page 5: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/5.jpg)
How it works
• Captures the wire traffic
• Follows TCP streams, decodes HTTP, MySQL, PgSQL, REDIS, Thrift-RPC
• Looks for requests, waits for the matching response
• Records response time, URLs, response codes, etc
![Page 6: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/6.jpg)
Show me! ( ̄^ ̄)
![Page 7: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/7.jpg)
![Page 8: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/8.jpg)
What do we do with the data? ¯\(°_o)/¯
![Page 9: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/9.jpg)
The traditional way
• Decide what metrics you need (requests per second for each server, response time percentiles, etc.)
• Write code to extract these metrics, store them in a DB
• Store the transactions in a DB
• Drilling down is difficult
• Features like “Top 10 method with errors” are difficult to implement
![Page 10: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/10.jpg)
PacketBeat + ELK
![Page 11: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/11.jpg)
Why ELK?
• Already proven to scale and perform for logs
• Clear and simple flow for the data
• You don’t have to pre-create the metrics
• Ad-hoc troubleshooting and analytics by using Kibana
• Drilling down to the problematic transactions is trivial
• Top N features are trivial
• Slicing by different dimensions is easy
![Page 12: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/12.jpg)
Show me! ( ̄^ ̄)
![Page 13: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/13.jpg)
![Page 14: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/14.jpg)
![Page 15: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/15.jpg)
![Page 16: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/16.jpg)
![Page 17: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/17.jpg)
Pros of wire data
• Captures a lot of things that other approaches miss
• No changes to the code or to the monitored application
• Minimal knowledge about the monitored app is required
• No latency overhead
• When using tap points, zero CPU/memory overhead on the app servers
![Page 18: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/18.jpg)
Cons of wire data
• There can be, like, tons of data
• Compared to log processing, larger CPU requirements
• Privacy concerns
• Doesn’t work for encrypted protocols
• Doesn’t work for “in-house” protocols
![Page 19: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/19.jpg)
Next steps ( ͡° ͜ʖ ͡°)
![Page 20: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/20.jpg)
More protocols
• Available:
• HTTP
• MySQL
• PostgreSQL
• REDIS
• Thrift-RPC
• Soon (tm):
• DNS
• Memcache
• MongoDB, RethinkDB
• Oracle, MSSQL
• XMLRPC / JSONRPC
• Your suggestions?
![Page 21: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/21.jpg)
Sampling
• Wire data can be huge
• Troubleshooting convenience vs hardware requirements
• Sample by:
• protocol (e.g. store all MySQL requests, sample REDIS 1/10)
• method (e.g. store all PUTs requests, sample GETs 1/10)
• status code (e.g. store all errors, sample successes)
• response time (e.g. store all slow transactions)
![Page 22: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/22.jpg)
String obfuscation
• Replace: select * from users where username=“Tudor” and id=3
• With: select * from users where username=S8 and id=N3
• Makes TopN charts better
• “The Mature Optimisation Handbook” - Carlos Bueno
![Page 23: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/23.jpg)
Bonito
• Our own UI
• Similar to Kibana, but focused more on app performance
• Will be a Kibana 4 plugin
![Page 24: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/24.jpg)
![Page 25: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/25.jpg)
Deploying PacketBeat (´ ▽`).。o♡
![Page 26: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/26.jpg)
Deploying
• Getting started guide
• packetbeat-deploy
• ansible roles for Packetbeat, Elasticsearch, Logstash, Redis, Kibana
• supports multiple ES nodes or all-in-one server
• ansible-playbook -i hosts site.yml
![Page 27: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/27.jpg)
Thanks! ( ゚▽゚)/
![Page 28: OSDC 2015: Tudor Golubenco | Application Performance Management with Packetbeat, Elasticsearch and Kibana](https://reader031.vdocuments.site/reader031/viewer/2022032022/55a5fba71a28abcd738b45e8/html5/thumbnails/28.jpg)
Keep in touch
• Twitter: @packetbeat or @tudor_g
• www: packetbeat.com
• github.com/packetbeat/packetbeat