calibration of performance test tooling roland van leusden
DESCRIPTION
Calibration of Performance Test tooling Roland van Leusden. Introduction. Senior Automation Specialist. 10+ years of experience in TA&PT. Passion For Technology. e: [email protected] m: 06 - 30 143 539. Contents. What triggered this investigation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/1.jpg)
Calibration of Performance Test tooling
Roland van Leusden
Proven Quality
![Page 2: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/2.jpg)
2
e: [email protected]: 06 - 30 143 539
10+ years of experience in TA&PTSenior Automation
Specialist
Proven Quality
Introduction
Passion For Technology
![Page 3: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/3.jpg)
Proven Quality3
Contents
• What triggered this investigation• Research definition & Scope• First Findings• Testenviroment setup• Testplan & Scenario• Results• Calibration
• Reference• Tool Behaviour & Solutions
• Results after calibration• Conclusion• When to calibrate ?• Next Steps
![Page 4: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/4.jpg)
Historical International Prototype Metre bar, made of an alloy of platinum and iridium, that was the standard from 1889 to 1960
Do performance test tools emulate the exact same behavior a real user would show when accessing the application ?
4
What triggered the investigation
Q:Proven Quality
![Page 5: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/5.jpg)
Scope: • 4 commonly used test tools, installed with default settings• Mix of commercial and Open Source tools• 1 virtual user to execute the test• 1 Browser• 1 OS
Goal:• Identify differences.• Adjust settings, calibration.
5 Proven Quality
Research Definition
![Page 6: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/6.jpg)
6 Proven Quality
First Findings
•Tooling doesn’t emulate browser behaviour like javascript execution.
•Different behavior at network level, number of tcp sessions
•Different behavior in requests methods
and number of requests.
![Page 7: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/7.jpg)
Server: Bootable Linux CDTurnkey Linux & OsCommerce
VMWare XP images:• Tool A• Tool B• Tool C• Tool D• Clean image
http://www.turnkeylinux.org/oscommerce
7 Proven Quality
Test Environment Setup
![Page 8: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/8.jpg)
1. Execute the testscenario manually, this is the Reference.2. Record the scenario with the testtool.3. Run the scenario with the testtool.
Scenario:• Add the dvd “Speed” on the front page to the cart• Search for a dvd with the word: “Mary”• Add dvd “There’s something about Mary” to the cart• Checkout• Create a new account• Complete the checkout & shipping• Return to the frontpage and logout
8 Proven Quality
Test Plan & Scenario
![Page 9: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/9.jpg)
Manual Tool A Tool B Tool C Tool D
Startpage 0,815 1,715 0,669 0,912 0,875Select DVD Speed 0,942 0,99 0,473 0,946 0,313Add to cart 0,197 2,071 3,21 0,14Search "Mary" 5,98 4,689 1,22 5,984Add to cart 1,367 6,92 2,092 3,538 0,094Check Out 0,421 1,715 0,25 0,897 0,078New Customer 0,408 0,637 0,896Account created 0,389 0,795 0,6 0,798 1,297Continue 0,498 0,156Shipping 0,374 0,13 0,417 0,883 0,312Payment 0,431 0,358 0,787 0,141Confirm Order 0,488 1,92 0,64 0,909 0,266Startpage 0,418 0,383 0,808 0,188Log Off 0,542 0,72 0,444 0,829 3,547
13,073 15,102 13,723 16,633 13,391
9 Proven Quality
Results of the test
Results are average of 10 runs.
![Page 10: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/10.jpg)
Proven Quality10
Calibration
• Create the reference by analysing user behaviour with the application.
• Analyse tool behaviour with the application.• Calibrate tool.• Validate calibration by rerunning the test.
![Page 11: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/11.jpg)
ManualNetwork Onload
Startpage 766ms 815ms
11 Proven Quality
Reference with FireBug
Use the profiler to determineJavascript execution time
![Page 12: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/12.jpg)
12 Proven Quality
Reference with WireShark
![Page 13: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/13.jpg)
5 sessions13 Proven Quality
Reference TCP Sessions
![Page 14: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/14.jpg)
Proven Quality14
Tool behaviour and Solutions
![Page 15: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/15.jpg)
Problem identified: Tooling doesn’t do processing like a browser. (javascript, rendering)
Solution: Compensate in the tool for clientside processing and if needed during a loadtest execute the scenario manually to capture the user experience.
15 Proven Quality
Compensate Browser processing
![Page 16: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/16.jpg)
Reference Tool A Tool B
16 Proven Quality
Post en Get Requests
![Page 17: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/17.jpg)
Problem identified: Different number ofRequests, GET and POST
17 Proven Quality
Adjust get & post request setting
Possible Solution: Adjust settings on how to Retrieve resources
![Page 18: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/18.jpg)
Proven Quality18
With Browser caching Without browser caching
Compensation for cached content using an “Only once Loop Controller”.
Adjust for Caching
Cached_Content_StartPage.csv
Problem identified: Some content is cached within the browser. Solution: Retrieve cached content only once for every VU.
![Page 19: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/19.jpg)
Reference Tool
19 Proven Quality
Network behavior
![Page 20: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/20.jpg)
Reference 5 sessions Tools 8 ~ 32 sessions20 Proven Quality
TCP Sessions
![Page 21: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/21.jpg)
Problem identified: Different behavior at network level caused by different number of connections used
21 Proven Quality
Adjust settings for connections
Possible Solution: Adjust settings for number of connections per virtual user
![Page 22: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/22.jpg)
Proven Quality22
Results after Calibration
Before Calibration
Manual Tool A Tool B Tool C Tool D
13,073 15,102 13,723 16,633 13,391
After Calibration
13,073 13,658 13,252 14,276 13,221
![Page 23: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/23.jpg)
The behavior and reported results from Performance Testtools need to be:
23 Proven Quality
Conclusion
VALIDATED & CALIBRATED
![Page 24: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/24.jpg)
Proven Quality24
When to Calibrate ?
• When the tool does partially / not support client behavior• Performance testing with more then 50 VU.• Most used / performance critical scenario.• No tool selection process has been done, available tooling is used.
Performance testing & calibration works very well in an Agile development process
![Page 25: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/25.jpg)
25 Proven Quality
How do we do this?
At Squerist part of our intake and tool selection process for performance testing is the calibration and validation of the tools we are going to use.
We monitor developments in the market on tooling, network and application level and conduct our own research.
![Page 26: Calibration of Performance Test tooling Roland van Leusden](https://reader035.vdocuments.site/reader035/viewer/2022062309/568145bd550346895db2c817/html5/thumbnails/26.jpg)
Are we finished? We want to research the influence of:
- Different browsers- Non web applications- Different OS- Different amounts of VU
On performance results and user experience
26 Proven Quality
Next steps
Why?
NO!!!
Because we are testers and our customers deserve accurate results!!!