model based software testing
DESCRIPTION
TRANSCRIPT
Esin Karaman
SWE550
Boğaziçi University
1
Agenda Software testing process evalution
What is model-based testing
The process of model-based testing
MBT (Model-based testing) Tools
Prerequisites for MBT
Benefits of MBT
Limitations of MBT
2
«Life is too short for manual testing.»
Harry Robinson, test architect for Microsoft’s Engineering Excellence Group and a driving force behind Microsoft’s model-based testing initiative, 1999-2004
3
Software testing process evalution
4
Manual Testing
Capture/Replay
Script Base Testing
Keyword-Driven Automated Testing
Model-Based
What is model-based testing? Model-based testing is a variant of testing for
automatic generation of test cases using models describing some aspects of the software under test (SUT).
It can automate the complete test design process, given a suitable model, and produces complete test sequences that can be transformed into executable tests cripts.
Model-based testing is the automation of the design of black-box tests
5
What is model-based testing? The difference from the usual black-box testing is that
rather than manually writing tests based on the requirements documentation, we instead create a model of the expected SUT behaviour, which captures some of the requirements. Then the model based testing tools are used to automatically generate tests from that model.
6
The process of model-based testing
Create a model detailed enough to describe the characteristics that we want to test
Automatically generate abstract test cases from the model by using a model-based testing tool
Transform abstract test cases into into executable test scripts
Execute test scripts by a test execution tool
7
8 Model-Based Testing Process
Modelling
A model of the SUT is built on the grounds of requirements
Includes expected behaviour
Must be concise and precise
9
Modelling (2)
Model-based testing uses models of the SUT and its environment.
Abstraction
- Function
- Data
M1-M3 is suitable
10
Modelling (3)
Modelling notations
State based (or Pre/Post) Notations
Transition-based Notations
Functional Notations
Operational Notations
Stochastic Notations
Data-Flow Notations
11
Test requirements selection
Used to control the generation of tests Steps:
Test selection criteria are defined
Test selection criteria are then transformed into test case specifications
12
Test selection criteria
Defining «best» test selection criteria is not possible. Test engineer chooses adequate test selection criteria. Criteria types:
Structural model coverage criteria
Data coverage criteria
Requirements-based coverage criteria
Ad-Hoc test case specifications
13
Test generation
Once the model and the test case specification are defined, an abstract test suite is generated.
Technology:
Random
Graph-search algorithms
Model-checking
Symbolic execution
Theorem proving
14
Test concretization(transformation)
Concretize abstract test suite to executable level (test script).
This can be done by a MBT Tool using some translation tables supplied by the test engineer.
The resulting executable tests may be Junit in Java, or in a dynamic language such as Tcl or Python, or in a dedicated test scripting language.
15
Test execution
Test scripts are executed and outcomes are compared to expected ones.
Tests are marked as pass, fail or inconclusive
Two approaches:
Online
Offline
16
Test execution(2)
Online Testing
Tests are executed as they are generated
MBT tool is tightly coupled to the SUT
Slow
Offline Testing
Tests are generated first, executed afterwards
Decouples the generation and execution phases
It can be used repeatedly for regression purposes
17
MBT Tools TorX
LTG (LEIRIOS Test Generator)
Matlab Simulink V&V
JUMBL
AETG
18
Prerequisites for MBT Someone enthusiastic
Test execution maturity
Modelling skills
Access to training, tools, and experts
The right kind of application
19
Benefits of MBT
Fault detection : MBT founds roughly the same number of errors as manually designed test suits
Reduced testing cost and time
Improved test quality
Requirements defect detection
Requirements evolution
20
Limitations of MBT
The model designers must be expert in the application area. This may require some training costs.
Usually used only for functional testing, little experience for other kinds of testing.
Sophisticated approach to testing, so it requires a reosonable level of test maturity. It is unwise to start using MBT without some experience with automated test execution.
21
References Mark Utting, Alexander Pretschner and Bruno Legeard
, A Taxonomy of Model-Based Testing, 2006
Bruno Legeard, Mark Utting, Practical Model-Based Testing A Tools Approach, 2007
A. Pretschner, W. Prenninger, S. Wagner, C. Kuhnel, M. Baumgartner, B. Sostawa, One Evaluation of Model-Based Testing and its Automation, 2005
22