Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. August 2006© 2006 Atos Origin. Confidential information owned by Atos Origin, to be used by the recipient only. This document or any part of it, may not be reproduced, copied, circulated and/or distributed nor quoted without prior written approval from Atos Origin.

Tester needed? No thanks, we use MBT!

EuroSTAR, The Hague, 12 November 2008

By Elise Greveraars

Atos Origin Nederland BV High Tech Campus 525656 AG Eindhoven - The Netherlands www.atosorigin.com

2/34

Agenda

» Introduction Model Based Testing

» Case studies & Lessons learned

» Future work

» Recommendations & conclusion

Scope: Administrative systems

3/34

Good morning, let’s wake up first!

» Have you ever..

» Created models before?

» Used models to design testcases?

» Used models to automatically generate testcases?

Many testers test model based

4/34

Introduction

» Informal Model Based Testing

» Formal Model Based Testing

5/34

Informal MBT: Introduction

Modelling tool

Risk Analysis

1. Create models

Requirements

2. Evaluaterequirement

s and models

Reviewresults

3. Design test cases

Test management tool

Manual test cases

4. Execution test cases

Test results

Test results

6. Execution Test cases

5. Administer test cases

Automated test cases

Test automation

tool

4. Adapts script for automation +

add physical test data

6/34

Advantages/disadvantage informal MBT

» Advantages» Easy to create» Early detection of specifications defects» Early insight into the application » Test model is guideline for design test cases» No costs for test generation tool

» Disadvantages» High initial costs on modelling» Manual work on test case design» No standardized test case design» Maintenance on models and test cases

7/34

Formal MBT: Introduction

(UML) Modelling tool

Risk Analysis

1. Create formal models

Requirements

3. Evaluate models

MBT solution tool

7. Administer test cases

Automated test cases

Test automation tool

6. Adapts script for automation +

map physical test data

Testmanagement toolManual testcases +

frame automated test cases

4. (Re) generate logical test cases

5. Import testcases

2. Evaluate requirements

Reviewresults

6. Map physical test data

Test results

8. Execute Test cases7. Execute Test cases

Test data

8/34

Advantages/disadvantages formal MBT

» Advantages» Early detection of specifications defects» Early insight into the application» Time saving on test case design» Standardized test case design» High test coverage

» Disadvantages» Extra costs for MBT tooling » High initial costs on modelling» Additional skills are needed» Mainly for functional/robustness testing

9/34

Position of formal MBT in V-Model

User requirements

System requirements

Acceptance test

System integration test

Global design Component

integration test

Component test

System test

Detailed design

Implementation

Functional domain

Technical domain

Problem domain

Solution domain

System requirements

System test

System integration test

Acceptance test

10/34

Model Driven Engineering (MDE) versus MBT

MDE:

MDE uses models for engineering purposes

» MDE» Purpose is to construct code» Technical models» Includes implementation information

» MBT» Purpose is to construct testcases» Functional models» No implementation information

MBT:

MBT uses models for testing purposes

11/34

MDE/MBT: To reuse or not to reuse models

» Test models from scratch» No (formal) models are available» Full independence» Fully customized models

» Partial reuse of engineering models » Strip irrelevant information» Partial independence » Model should be customized

» Reuse of reversed models » Good for regression testing » Basic set of models to be adjusted» No independence» Model should be customized

» Combination of creating and reusing

12/34

Formal MBT: Guidelines modelling

» Choose a model notation (for example UML) + MBT tool

» Choose the risks you want to model

» Model only functional information

» Model the expected behaviour, models should be:» Accurate» Reasonable sized » Easy to maintain

» Use parameters to represent data

13/34

Formal MBT: Test generation techniques

» Selection criteria to generate test case

» Selection criteria depends on type of model

» Example of selection criteria for state diagram» State coverage» Transition coverage» All transition pairs

» Selection criteria determine:» Depth of testing» Size of test suite

State 3 State 5

State 1

S1

S2

S5

S4S3

AB

C

D

E

G

F

State 3 State 5

State 1

S1

S2

S5S5

S4S4S3

AB

C

D

E

G

F

14/34

Current market

» Tools

» Market adoption

15/34

Current market development formal MBT: Tools

» Support generation of test cases

» Some support modelling

» Some support test execution

» Support of various:» Model notations» Test generation techniques» Test case output

» Suppliers MBT tooling:SmarTesting, Verisoft Technology, All4tec, etc

16/34

Current market development formal MBT: Adoption

17/34

Agenda

» Introduction Model Based Testing

» Case studies & Lessons learned

» Future work

» Recommendations & conclusion

18/34

Case studies

» Case study on Informal Model Based Reviewing

» Formal MBT

19/34

Case study on informal Model Based Reviewing

Case study settings:» 2 versions of a newly built application» Informally reviewed» Same functional designer

Version 1: 104 function points, no models usedVersion 2: 113 function points, model used

Version 1 Version 2

20/34

Lessons learned Model Based Reviewing

» Model Based Reviewing very useful

» Missing and unclear specifications found

» Testers gained much knowledge

» Create models during reviewing sessions

» Less commitment by designers in later phase

21/34

Case study on formal model based testing

» Team was trained in 3 weeks on formal modelling and tools

» Incremental & iterative approach

» Order intake application

22/34

Case study: Used models

» Class diagram

» Object diagram

23/34

» State diagram & Object Constraint Language (OCL):

Case study: Used models

24/34

Case study: generated testcases

» Coverage criteria used:» Each transition » Conditions in the decisions (D/CC criteria) » One data value per equivalence class

25/34

Lessons learned formal MBT

Pro’s

» Models provide a better overview then text

» Traceability possible

» Maintenance shifts to modelling

» Combine MBT & test automation

» Modelling is a very creative task

» New exiting challenge and fun for testers!!

Cons

» Models not business user friendly

» No risk based test case generation yet

» Room for improvement on readability

26/34

Lessons learned: Competence

New role needed to additional roles: Testconstructor

Needs knowledge on:» Transforming written specifications into models » Creating models » Testing» Domain » Programming

27/34

Lessons learned: Competence

Traditional roles:

» Gain knowledge on» MBT and tooling» Modelling and tooling

» Test manager/coordinator» Adapt test strategy

» Test automate» Explore MBT tooling» Explore generated test cases

» Test designer» Design test cases not covered by MBT

» Test executer » Execute manual test cases

Hop on board and get enthusiastic!!

28/34

Agenda

» Introduction Model Based Testing

» Case studies & Lessons learned

» Future work

» Recommendations & conclusion

29/34

Future work

Risk based test generation

» Test manager in control on test case generation» High level risks -> Higher coverage criteria» Low level risks -> Lower coverage criteria

» Test generation based on probability

Path A-B: 15%

Path A-E-F-G: 25%

Path C-D-G: 60%

State 3 State 5

State 1

S1

S2

S5

S4S3

AB

C

D

E

G

F

State 3 State 5

State 1

S1

S2

S5S5

S4S4S3

AB

C

D

E

G

F

30/34

Future work

» Business model based testing» Business friendly models + test cases» Simulation possible

User requirements

System requirements

Acceptance test

System integration test

Global design Component

integration test

Component test

System test

Detailed design

Implementation

Functional domain

Technical domain

Problem domain

Solution domain

Business requirements Business

requirements

User requirements

Acceptance test

31/34

Agenda

» Introduction Model Based Testing

» Case studies & Lessons learned

» Future work

» Recommendations & conclusion

32/34

Recommendations

» Use informal MBT

» Use formal MBT when regression tests are expected

» Educate a test constructor

» Start small to gain experience

» Create models based on risks

» Start to model as early as possible

» Keep the models simple

33/34

Conclusion

Tester still needed? Yes for sure!!

34/34

Questions?

More information on this presentation:

- Eurostar CD

- Mailto: MBT@atosorigin.com

- Contact:Elise GreveraarsMobile: +31 (0) 6 52594072elise.greveraars@atosorigin.com