performance testing in - cloudbees · pdf fileperformance testing in continuous delivery...
TRANSCRIPT
![Page 1: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/1.jpg)
Performance Testing in Continuous Delivery Pipelines
Andrey PokhilkoChief Scientist, BlazeMeter
![Page 2: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/2.jpg)
Importance of CI and CD
● Machine time costs nothing, human time priceless
● De-facto winning practice
● Most advanced teams go with CD
● A lot of teams are still in process of adopting it
![Page 3: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/3.jpg)
Agenda
1. Things we put into CI
2. Best practice (for now)
3. Challenges of testing in CI
4. Jenkins for Performance Testing
![Page 4: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/4.jpg)
Things we put into CI
![Page 5: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/5.jpg)
Triggering and Preparations
● When to do the job
● VCS checkout + dependencies checkout
● Building project (compiling etc.)
● Put resulting packages into repos
![Page 6: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/6.jpg)
Quality Control
1. Static code analysis
2. Unit tests
3. Functional tests
4. Performance testsneeds deployment
![Page 7: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/7.jpg)
Deployment
● To QA environment (for further manual tests)
● To staging environment
● To production environment
![Page 8: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/8.jpg)
The Best Practice
![Page 9: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/9.jpg)
VCS Checkout + Dependencies
Build Unit Tests
Functional Tests
Perfromance Tests
Deployment
Filter Filter Filter Filter
One Way Road (Driverless)
![Page 10: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/10.jpg)
VCS-Driven Pipeline
● Natural evolution of CI systems
● Branching and pull requests
● Jenkins 2.0 pipelines
● Taurus Tool as part of this approach
![Page 11: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/11.jpg)
Why testing is challenging in CI
![Page 12: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/12.jpg)
Challenge #1: Test Environment
● Applications are complex
● Lots of dependencies
● Third-party systems
Databases
Microservices
Third-parties
![Page 13: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/13.jpg)
Challenge #2: Time Consuming
● Preparations
● A lot of functional tests
● Performance tests are naturally long
![Page 14: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/14.jpg)
Challenge #3: Debugging CI Jobs
● Evolving test complexity
● Debugging and troubleshooting
● Build history is a value
![Page 15: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/15.jpg)
Challenge #4: Results Analysis
● Rich reporting needed
● Jenkins UI has its limits
● Automated decision making for CD
● Collaboration for non-tech people
wut???
![Page 16: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/16.jpg)
Overcoming Challenges
![Page 17: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/17.jpg)
Overcome: Test Environment
● Containers help (you have to use containers :)
● You don’t need to be as realistic
≈
![Page 18: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/18.jpg)
Overcome: Time Consuming
● Prepare what you can upfront - in the night
● Reuse what you can
● Short tests can reveal a lot of things
● Don’t try to put spike and endurance tests into CI
● Parallelize tests, Jenkins 2.0 or Taurus helps
![Page 19: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/19.jpg)
Overcome: Debugging and Improving
● Taurus Tool, the handheld piece of CI
![Page 20: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/20.jpg)
Overcome: Debugging and Improving
Load Test Job
Local Debugging
![Page 21: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/21.jpg)
Overcome: Reporting & Result Consumption
● Purpose-built services with integrations
● Overview & Status in Jenkins
![Page 22: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/22.jpg)
Overcome: Decision Making
● Deploy to staging is great win
● Have pass/fail criteria, including APM info
● It’s ok to have deploy as separate job
(for UAT, to avoid disruptions)
![Page 23: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/23.jpg)
Jenkins for Performance Testing
![Page 24: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/24.jpg)
Why Use Jenkins for Testing?
● Multi-step process
● Long process
● Repetitive process
![Page 25: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/25.jpg)
Start Simple
Load Test 1
Load Test 2
Load Test 3
Center of Excellence
Developer
Operations
QA
Maintain Jobs
![Page 26: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/26.jpg)
Become Continuous
● Periodic jobs have many applications
● Have valid failure criteria (thresholds)
● Use non-blocking downstream jobs
![Page 27: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/27.jpg)
Pursue Continuous Delivery
FastFeedbackDeveloper /
DevOps
Build + xUnitFunctional
Tests
Load Tests Deploy Staging
SlowerFeedback
![Page 28: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/28.jpg)
Some Useful Plugins
![Page 29: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/29.jpg)
Plot Plugin
![Page 30: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/30.jpg)
Performance Publisher Plugin
![Page 31: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/31.jpg)
BlazeMeter Plugin
![Page 32: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/32.jpg)
Taurus Tool
● Why not plugin, but command-line tool
● Universal plug for tests inside Jenkins
● Taurus designed to work with other Jenkins plugins
gettaurus.org
![Page 33: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/33.jpg)
Final Summary
1. We have to test inside CI
2. There are ways to start it simple
3. There are tools & plugins to help
4. Jenkins 2.0 pipelines FTW!
![Page 34: Performance Testing in - CloudBees · PDF filePerformance Testing in Continuous Delivery Pipelines Andrey Pokhilko Chief Scientist, BlazeMeter](https://reader034.vdocuments.site/reader034/viewer/2022051508/5aa634147f8b9ac8748e223b/html5/thumbnails/34.jpg)
Thank you!Any questions?