![Page 1: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/1.jpg)
Untersuchung der Prozessgrenzen und Wärmeakkumulationbei der Präzisionsbear-beitung von Metall mit ultrakurz gepulster Laserstrahlung
10th ACM/SPEC Int. Conf. on Performance Engineering (ICPE 2019)
Mumbai, India – April 11, 2019
Behavior-driven Load Testing Using Contextual Knowledge– Approach and Experiences –
Henning Schulz, Dušan Okanović,
André van Hoorn
Vincenzo Ferme, Cesare Pautasso
@andrevanhoorn
![Page 2: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/2.jpg)
Load Testing – Approach and Challenges
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 3
Workload
specification
SUTLOAD
Test results
Load
driver
System
under Test
Expert
High manual effort for maintaining load tests
Load tests need much time to execute
There are no suitable load tests
![Page 3: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/3.jpg)
BenchFlow – Declarative Load Testing
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 4
Background
SUTLOAD
Workload
specification
(declarative)
Load
driver
System
under Test
BenchFlow
SUT definition
Test results
• Generation of executable artifacts
• SUT deployment
• Load test execution
• Collection of test results
Ferme et al., ICPE 2018
ExpertExample BenchFlow specification
Manual workload specification needed.
![Page 4: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/4.jpg)
ContinuITy – Continuous Load Testing in DevOps
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 5
Background
Monitoring data
Workload
specification
SUTLOAD
Test resultsContextual
information
ContinuITy
Load
driver
System
under Test
• Workload model extraction
• Workload model evolution
• Workload prediction
• Workload model selection
Schulz et al., LTB@ICPE 2018
workload
contextual
knowledge
Black Friday Black Friday
Christmastime
past future
Workload – prediction and context
Limited load test execution automation.
![Page 5: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/5.jpg)
Behavior-driven (Functional) Testing – Workflow
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 6
Background
ThenWhenGiven
Examples ©
Initial context
Changes
Outcome
![Page 6: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/6.jpg)
Behavior-driven Load Testing (BDLT) – Example
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 7
Approach
BDLT definition
then
when
Given
BDLT example definition
Given the next Black Friday,
when varying the CPU cores between 1 and 4,
then run the experiment for 1h
andensure the maximum CPU utilization is less than 60%.
thenoutcome
quality gates
run time
metrics
whenchanges
num. users
event
app config
Giveninitial context
event
app config
date (range)
![Page 7: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/7.jpg)
Behavior-driven Load Testing (BDLT) – Workflow
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 8
Approach
Monitoring data
Workload
specification
SUTLOAD
Test resultsContextual
information
ContinuITy
Workload
specification
(declarative)
Load
driver
System
under Test
BenchFlow
BDLT definition
Then
When
GivenSUT definition
![Page 8: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/8.jpg)
BDLT Language Definition
EBNF-based BDLT grammar with extension points for events
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 9
Giveninitial context
event
app config
date (range)
Whenchanges
num. users
event
app config Thenoutcome
quality gates
run time
metrics
![Page 9: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/9.jpg)
Mapping from BDLT to ContinuITy and BenchFlow
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 10
![Page 10: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/10.jpg)
Evaluation – Research Questions
How expressive is the BDLT language in regards to load test concerns of industrial use cases?
RQ1
How would BDLT be used in industrial contexts?RQ2
What are the benefits and limitations of using BDLT in comparison to defining load test scripts?
RQ3
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 11
![Page 11: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/11.jpg)
Case Study – System and Method
● System under study
● IoT system from automotive sector
● IoT endpoint migrating to Cloud
● Methodology
○ Workshops
○ We iteratively developed test specifications based on expert feedback
○ Incorporation of production monitoring data (per-device behavior and intensity)
● Four scenarios
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 12
![Page 12: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/12.jpg)
Case Study – Configuration Exploration
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 13
Given the next three monthsandthe number of users set to the maximum
when varying the CPU cores
between 0.5 and 4 in steps of 0.5andvarying the number of instancesbetween 1 and 5
andvarying the RAM among (1GB, 2GB, 4GB)
then run each experiment for 1 hourandensure the average CPU load is less than15%
andensure the message latency is less than2 seconds
![Page 13: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/13.jpg)
Case Study – Recovery Spike
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 14
Given 2018/10/15 9:00
when an outage happened from2018/10/15 7:00 to 2018/10/15 9:00
then run the experiment for 2 hours
and
ensure the final queue length isless than 100
![Page 14: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/14.jpg)
Evaluation – Lessons Learned
How expressive is the BDLT language in regards to load test concerns of industrial use cases?
RQ1
How would BDLT be used in industrial contexts?RQ2
What are the benefits and limitations of using BDLT in comparison to defining load test scripts?
RQ3
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 15
Could express all use cases
Custom extensions needed
Could replace manual tests
Natural language helps understanding
and communication (with non-experts)
“The approach has potential“
Some concerns hard to express
Current (technical) limitation to HTTP
Helps identifying new concerns
Too many (technical) details in the Then clause?
![Page 15: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/15.jpg)
+ Additional (laboratory) case study
+ Supplementary material
• Extending the language
• Improving tool support
• Further evaluation
• Supporting regression queries
• Integration into agile development
methods, e.g., user stories from
Scrum tickets
• Integration into Declarative Performance
Engineering landscape
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 16
Summary Future Work
Walter et al., ICPE 2016
Avritzer et al.,ECSA 2018
![Page 16: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/16.jpg)
Untersuchung der Prozessgrenzen und Wärmeakkumulationbei der Präzisionsbear-beitung von Metall mit ultrakurz gepulster Laserstrahlung
10th ACM/SPEC Int. Conf. on Performance Engineering (ICPE 2019)
Mumbai, India – April 10, 2019
Behavior-driven Load Testing Using Contextual Knowledge– Approach and Experiences –
Henning Schulz, Dušan Okanović,
André van Hoorn
Vincenzo Ferme, Cesare Pautasso
@andrevanhoorn
![Page 17: Behavior-driven Load Testing Using Contextual Knowledgedesign.inf.unisi.ch/sites/default/files/talks/20190410-ICPE-BehaviorDriven-16zu10...10th ACM/SPEC Int. Conf. on Performance Engineering](https://reader033.vdocuments.site/reader033/viewer/2022042809/5f91385ee1b7501fa60c51a9/html5/thumbnails/17.jpg)
References
Alberto Avritzer, Vincenzo Ferme, Andrea Janes, Barbara Russo, Henning Schulz,
André van Hoorn: A Quantitative Approach for the Assessment of Microservice
Architecture Deployment Alternatives by Automated Performance Testing. ECSA
2018: 159-174
Vincenzo Ferme, Cesare Pautasso: A Declarative Approach for Performance Tests
Execution in Continuous Software Development Environments. ICPE 2018: 261-272
Henning Schulz, Tobias Angerstein, André van Hoorn: Towards Automating
Representative Load Testing in Continuous Software Engineering. ICPE Companion
2018: 123-126
Christian Vögele, André van Hoorn, Eike Schulz, Wilhelm Hasselbring, Helmut
Krcmar: WESSBAS: extraction of probabilistic workload specifications for load
testing and performance prediction - a model-driven approach for session-based
application systems. Software and System Modeling 17(2): 443-477 (2018)
Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, Samuel Kounev:
Asking "What"?, Automating the "How"?: The Vision of Declarative Performance
Engineering. ICPE 2016: 91-94
Ferme et al., ICPE 2018
Schulz et al., LTB@ICPE 2018
Walter et al., ICPE 2016
Avritzer et al.,ECSA 2018
Vögele et al., SoSyM 2016