pushing prometheus until it breaks. krasi georgiev, harsh ... · openshift kiali operations...
TRANSCRIPT
![Page 1: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/1.jpg)
Prometheus
Pushing Prometheus until it breaks.The bumpy road to a fully automated benchmarking.
Krasi Georgiev, Harsh Agarwal
@krazygeorgiev@thesipian
![Page 2: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/2.jpg)
Prometheus
Climbing and Carrots
no problem if you pronounce it “crazy”
Krasi Georgiev
github.com/krasi-georgiev
![Page 3: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/3.jpg)
Prometheus
Climbing and Carrots
no problem if you pronounce it “crazy”
Prometheus maintainer from March
Krasi Georgiev
github.com/krasi-georgiev
![Page 4: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/4.jpg)
Prometheus
Climbing and Carrots
no problem if you pronounce it “crazy”
Prometheus maintainer from March
Part of the Prometheus team.
Krasi Georgiev
github.com/krasi-georgiev
![Page 5: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/5.jpg)
Prometheus
● Openshift● Kiali● Operations monitoring
Prometheus in Red Hat
![Page 6: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/6.jpg)
Prometheus
● Undergraduate student from
● Google Summer of Code 2018 intern for Prometheus○ Mentors
■ Krasi Georgiev■ Goutham Veeramachaneni
Harsh Agarwal
@thesipian
github.com/sipian
![Page 7: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/7.jpg)
Prometheus
Is prometheus ready for a new release?
![Page 8: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/8.jpg)
Prometheus
I day of a Prometheus maintainer.
![Page 9: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/9.jpg)
Prometheus
Wake up and get ready for work.
![Page 10: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/10.jpg)
Prometheus
User: My Prometheus is broken please help!
Me: Hey man I am not a psychic , give some details.
User: 100 lines config scraping 3500 targets.
Me: ooooo sh*t…. replicate this locally…........?
Sounds Familiar?
![Page 11: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/11.jpg)
Prometheus
Why benchmarking?Why not just unit and e2e tests?
● Memory leaks appear at high load and happen over a long period.● Compaction can be tested only with long running tests.
![Page 12: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/12.jpg)
Prometheus
● Build by the k8s testing team and runs on k8s○ each job is a via pod deployments
● Integrates nicely with github○ triggers via github comments
● Written in golang● Easy to extend via plugins● Used by k8s, openshift, istio, jetstack
Extended discussions: https://goo.gl/CuKsMB
Why Prow CI?
![Page 13: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/13.jpg)
Prometheus
● Drone CI○ No comment triggers (WIP)○ I bit hacky to make it run jobs on k8s(WIP)
● Packet - Cloud on Bare Metal○ kudos to their amazing team for helping.○ decided to use GKE to avoid the k8s bootstrapping
● terraform , kubectl○ Prefer to troubleshoot few lines of golang code○ Life is too short to learn new tools :)○ The prombench tool is easy to use and no dependencies
Didn’t make the finals.
![Page 14: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/14.jpg)
Prometheus
Recorded Demo?
https://github.com/prometheus/prombench
![Page 15: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/15.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 16: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/16.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 17: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/17.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 18: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/18.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 19: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/19.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 20: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/20.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 21: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/21.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 22: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/22.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 23: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/23.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 24: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/24.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 25: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/25.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 26: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/26.jpg)
Prometheus https://github.com/prometheus/prombench
Life of aBench test
![Page 27: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/27.jpg)
Prometheus
Results example PR vs Master
![Page 28: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/28.jpg)
Prometheus
● Document how to run manually without prow.
● Catch Races with the -- race detector.
● Add Service Discovery tests.
● Automated Scalability tests.
○ post results with every release
● New cli commands
○ `promtool debug` - gather metrics/profile data
○ `tsdb scan` - scan and repair/delete corrupted blocks(WIP).
This is only the beginning!
![Page 29: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/29.jpg)
Prometheus
Positively surprised of how much support we received from everyone.
● Packet , Google , RedHat(for letting me work on this).● Goutham, Frederic , Max for providing feedback along the way ● Julius - for putting us in touch with the CNCF for the GKE credit.● Chris from the CNCF for setting up the google account.● Fabian & Max for their first Prombench version.
○ Many pieces were all ready to use.● The kubernetes-sig-testing team for introducing us to prow.
○ https://github.com/cjwagner○ https://github.com/krzyzacy
the Big thanks
![Page 30: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/30.jpg)
Prometheus
Conclusion!
If you don’t break = you don’t refactor and optimize.
![Page 31: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/31.jpg)
Prometheus https://github.com/kiali/kiali
Istio
Prometheus
Jaeger
Open source - Mesh observability
![Page 32: Pushing Prometheus until it breaks. Krasi Georgiev, Harsh ... · Openshift Kiali Operations monitoring Prometheus in Red Hat. Prometheus ... Build by the k8s testing team and runs](https://reader033.vdocuments.site/reader033/viewer/2022052019/6032d4abee54e76774194f55/html5/thumbnails/32.jpg)
Prometheus
Slides: https://goo.gl/ky5ZZX
Code: https://github.com/prometheus/prombench
Proposal: https://goo.gl/CuKsMB
If you DON’T have any questionsplease ask them now.
Questions for extreme sports are welcome :)