automatic test case generation
DESCRIPTION
Automatic Test Case Generation for Function Block DiagramsTRANSCRIPT
![Page 1: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/1.jpg)
Automatic Test Case Generation for Train Control Systems Adnan Causevic
TOCSYC Industrial Workshop, 2014-11-13, Västerås
![Page 2: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/2.jpg)
2
Today’s talk is about…
![Page 3: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/3.jpg)
● Software Testing Laboratory @ MDH
● Research: ● Test Automation ● Quality of Tests ● Developers Testing
● AGENTS - KKS funded project ● Bombardier Transportation ● Maximatecc
● Involved in the TOCSYC research environment
● Intelligent Embedded Systems Master Program Coordinator
3
About me
![Page 4: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/4.jpg)
● ATAC research project ● 2011-2014 ● Funded by ITEA2/Vinnova ● Advanced Test Automation for Complex Software-
Intensive System
4
Cooperation with Bombardier
“The project has brought test automation on the agenda of European industry.”
- ITEA3 Final Review Report
![Page 5: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/5.jpg)
● Train Control Management System (TCMS) ● a high capacity, infrastructure backbone ● the center of the distributed system
● Functions controlled by TCMS include ● collecting line voltage, ● controlling the train engines, ● opening and closing the train doors, ● upload of diagnostic data.
5
About Bombardier
![Page 6: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/6.jpg)
● Vehicle ● Vehicle Integration testing
● System ● System Integration testing
● Software ● Modules integration testing ● Modules testing
● “Get to the driver’s perspective as soon as possible”
6
About Bombardier … and their testing
Responsibility of a Developer!
![Page 7: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/7.jpg)
● IEC 61131-3 ● Function Block Diagram ● Graphical programming language ● Based on element composition ● Various predefined elements
7
Development Environment
![Page 8: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/8.jpg)
● Coverage ● Strict measurements mandated by safety standards
● Tooling ● No tool support to measure coverage on FBD
● Time ● Creating tests just to reach the coverage leaves very little
time for functional testing
8
Current challenges … with module testing
![Page 9: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/9.jpg)
9
CompleteTest
![Page 10: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/10.jpg)
10
Research approach FBD Program
Transformation 1 Annotation2UPPAAL timed
automata
...
Reachability properties
∃ ♢ β ,∃ ♢ pi ,
∃ ♢ c0 and c1
Test traces Test cases(Step) (1)(Time) (20s)(Inputs) (0 1 23)(Outputs) (1 2 543,9)
UPPAAL model checker
Test Generation3
Logic-‐‑based coverage criteria
DC, CC, MC/DC
”Automated Test Generation using Model-Checking: An Industrial Evaluation”, Eduard Paul Enoiu, Adnan Causevic, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, Paul Pettersson. International Journal on Software Tools for Technology Transfer, 2014, Springer.
![Page 11: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/11.jpg)
● Time required to generate tests ● satisfying the DC, CC and MC/DC logic coverage criteria
11
Case Study at BT
![Page 12: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/12.jpg)
● For 34 of the 157 programs, the tool did not terminate after running for a substantial period of time.
● Cut-off time was set to 10 minutes.
● There is an indication that as the number of decisions increases, the performance deteriorates and the cost of using the tool may become prohibitive.
● But, if you have so many decisions, how would you test it manually?
12
Case Study at BT (2)
![Page 13: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/13.jpg)
● Should a developer generate the tests from scratch? ● Should a developer provide manual tests first? ● How many (or how much in terms of coverage)?
● When should a developer use the Validate Test button?
● Should developers use this tool at all or should we automate that as well (nightly build, etc.) ?
13
Best Practices?
![Page 14: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/14.jpg)
● Internal behavior of each function block has to be modeled ● Currently, blocks from the standard language are modeled ● and a few custom made by Bombardier which are often used
● State-space explosion ● a known problem of model-checking ● large number of Boolean or Integer input values ● but, again, how would you do it manually ?
14
Limitations
![Page 15: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/15.jpg)
● Complementing model-checking with: ● Static analysis of FBD’s ● Search-based software testing
● Trying some new approaches: ● Different model-checkers, SMT solvers, …
● Measuring effectiveness of generated tests ● In terms of how good they are in finding faults
● Experimenting with FBD programs from
other organizations
15
On-going and Future work
![Page 16: Automatic Test Case Generation](https://reader034.vdocuments.site/reader034/viewer/2022042700/5594448d1a28ab01308b47d8/html5/thumbnails/16.jpg)
● Adnan Causevic [email protected]
● Eduard Enoiu [email protected]
● www.completetest.org
16
Questions?