integration testing of components guided by incremental state machine learning

Integration Testing of Components Guided by Incremental State Machine Learning

Keqin Li, CNRS LSR-IMAGRoland Groz, INPG LSR-IMAGMuzammil Shahbaz, France

TelecomAugust 2006

2

Content Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion

3

Motivation

Software development is more and more based on component integration

Unknown Internal structures and insufficient documents

Only partial and informal information is available

Provide a required system assembling COTS of which he has a limited knowledge

Unit testing and integration testing

Have to rely on intuitions and little support is available

Component: Integrator:

4

Our Approach Infer formal models of components

from behaviours observed during tests

Perform integration testing based on the formal models

Iteratively improve the formal models

5

Background - Angluin’s Algorithm [1987] Infer formal model (conjecture)

from observed behavior (observation table)

For deterministic finite-state acceptor

Teacher answers Membership query Equivalence query

6

Process

EmptyObservation

TableMembership

QueryClosed& Cons.

No

EquivalenceQuery

YesYes? End

Add strings toObservation Table

Yes

Teacher

String AcceptedOr Not

Conjecture Yes OrCounterexample

No

7

Assumptions Black boxes with known interfaces Only external interfaces are

controllable Slow environment Test scenarios

8

Framework

UnitTesting

Models ofComponents

IntegrationTesting

ScenarioTesting

InitialObservation

Tables

ConsultDomainExpert

9

I. T.C.E.

Models

M1(1)

M2(1)

…

Mn(1)

U. T.

U. T.

U. T.

I. T.

M1(2)

M2(2)

…

Mn(2)

U. T.

U. T.

U. T.

C.E.I. T.

M1(3)

M2(3)

…

Mn(3)

U. T.

U. T.

U. T.

U. T.:UnitTesting

I. T.:IntegrationTesting

C.E.:Counter-example

10

Scenario Testing A test scenario is an input/output

sequence containing external inputs and outputs

A test case is constructed for each test scenario

Replace problematic component Record observed behaviour of each

component in its observation table

11

Unit Testing Each Component is tested

individually For each component

Complete the initial observation table Using extended learning algorithm to

construct closed and consistent observation table

Make conjectured model

12

Integration Testing

TestGeneration

Discre-pency?

Yes

CoverageCriterionSatisfied?

No Yes

No

UnitTesting

TestExecution

13

Test Generation based on Composition of FSMs M N [Petrenko 98] An algorithm to compute M N

[Lima 97] Assign colour to “global”

transitions An algorithm to generate tests

covering colours [Kim 02]

14

References [Petrenko 98] A. Petrenko, N. Yevtushenko,

“Solving Asynchronous Equations,” Proceedings of FORTE’98, 1998

[Lima 97] L. Lima Jr., A. Cavalli, “A Pragmatic Approach to Generating Test Sequences for Embedded Systems,” Proceedings of IWTCS’97, 1997

[Kim 02] M. Kim, C. Besse, A. Cavalli, and F. Zaïdi, “Two Methods for Interoperability Tests Generation: An Application to the TCP/IP Protocol,” Proceedings of TestCom 2002, 2002.

15

Dealing with Discrepancy Observe the behaviour of each

component, and compare with its model

Discrepancy Unexpected output sequence New input symbol

16

Refine (relearn) model Counterexample or new input

symbol has been found Add corresponding strings into

observation table Perform learning algorithm to

make the observation table closed and consistent again

Make conjectured model

17

Consulting Domain Expert Models of components and test

results Domain expert

Identify unexpected behaviours Propose additional test scenarios

Iteration

18

Results of Integration Testing For each component, we have a

model The joint behaviour of these

components has been systematically tested

Faults might be revealed during testing

19

Conclusion Combines machine learning and

model-based testing for the integration of components

Reuses Angluin’s algorithm Iteratively refines models

20

The Models Just a “by-product” Our goal is not to build a complete

model of the system Just used as basis for the testing

process to: Generate integration tests of

components Assess the coverage, hence the

quality of the integration testing process

21

The Next Steps Move to Parameterized FSM

presentation by Muzammil Shahbaz Identify the type of approximation

needed Experiment and case study Revisit model-based test

generation strategies

Thank you

Backup

24

Observation Table

00bbb10bba00ba00ab01aa01bb00b10a01εaε

S

S.A

E

25

Closed

tSA, sS, s.t., row(t)=row(s)

00bbb10bba00ba00ab01aa01bb00b10a01εaε

S

SA

E

26

Consistent

s1S, s2S, row(s1)=row(s2),

Then, aA,

row(s1a)=row(s2a)

00bbb10bba00ba00ab01aa01bb00b10a01εaε

S

SA

E

27

Conjecture

00bbb10bba00ba00ab01aa01bb00b10a01εaε

S

SA

E

q0

q1 q2

a b

ab

a

b

28

Observation Table

xybbyxbaxyabyxaaxybyxaxxεba

row(ε)

row(b)row(a)

a/x b/x

b/y

a/ya/x b/x

29

Example – Component Hotel Reservation

q0

q2

q4

q3

q1

q5

q6

q7

q9

q8 q10

q11

N/HLH1/TL

Dbl/F

Sgl/F

T/PD1

T/PS1No/HL

No/TL

No/TL

No/F

No/F

OK/CD1

OK/CS1

H2/TL

Dbl/F

Sgl/F

No/TL

No/TL

T/PD2

T/PS2

No/F

No/FOK/CD2

OK/CS2

30

Initial Observation Table

ΩΩΩΩΩΩΩHLNoΩΩΩΩΩΩΩHLOKΩΩΩΩΩΩΩHLTΩΩΩΩΩΩΩHLSglΩΩΩΩΩΩΩHLDblΩΩΩΩΩΩΩHLH2ΩΩΩΩΩΩΩHLH1ΩΩΩΩΩTLTLΩNΩΩΩΩΩΩΩHLεNoOKTSglDblH2H1N

31

Add N to S

………………………ΩΩΩΩΩTLTLΩN/No

ΩΩΩΩΩΩΩHLNo

ΩΩΩΩΩTLTLΩN/DblHLΩΩFFΩΩΩN/H2HLΩΩFFΩΩΩN/H1ΩΩΩΩΩTLTLΩN/N

………………………ΩΩΩΩΩΩΩHLH1ΩΩΩΩΩTLTLΩNΩΩΩΩΩΩΩHLεNoOKTSglDblH2H1N

32

Conjecture

row(ε) row(N-H1)

row(N-H1-S)

row(N-H1-D)

row(N)

row(N-H1-D-T)

row(N-H1-S-T)

N/HL H1/TLD/F

S/F

T/PD1

T/PS1

No/HL

No/TL

No/TL

No/F

No/F

OK/CD1

OK/CS1

H2/TL

33

Example – Component User Interface

q0

UI_Name/N

q1HL/UO_HL

UI_H1/H1UI_H2/H2

q2TL/UO_TL

UI_Dbl/DblUI_Sgl/SglUI_No/No

q3F/UO_F

UI_T/TUI_No/No

q4

PD1/UO_PD1PS1/UO_PS1PD2/UO_PD2PS2/UO_PS2

UI_OK/OKUI_No/No

F/UO_FTL/UO_TLHL/UO_HL

CD1/UO_CD1CS1/UO_CS1CD2/UO_CD2CS2/UO_CS2

34

Integration Testing Environment

User Interface(M)

Hotel Reservation(N)

oMN

oNM

cM

oM

35

Counterexample UI_Name/UI_H2/UI_Dbl/UI_T The real input sequence of Hotel

Reservation is: N/H2/Dbl/T

The corresponding output sequence is: HL/TL/F/PD2

According to the model, the output should be: HL/TL/F/PD1

36

Comparing with Angluin’s Algorithm

InitialObservation

TableUnit Testing Closed

& Cons.

No

IntegrationTesting

Yes Counter-example? End

Add strings toObservation Table

Yes

No

ScenarioTesting Component Component

AComponent

B

Test Output Test Output