model-based testing - aalborg universitet · model-based testing 3. un-timed model-based testing...

Quasimodo1

Model-Based Testing

applied to a Wireless Sensor Network Node

Jan Tretmans

Embedded Systems Institute, Eindhoven, NL

and Radboud University, Nijmegen, NL

Quasimodo

with support from:Marcel Verhoef ChessFrits vd Wateren ChessAxel Belinfante University of TwenteFeng Zhu Radboud UniversityJulien Schmaltz Open University NL

Quasimodo2

Overview

1. The Challenge: A Wireless Sensor Network Node

2. Model-Based Testing

3. Un-Timed Model-Based Testing with Labelled Transition Systems

4. Real-Time Model-Based Testing with Timed Automata

5. Model-Based Testing Tools

6. Back to the Challenge: The Wireless Sensor Network Node

7. Demo

8. Concluding Remarks

Quasimodo3

The Challenge:

A Wireless Sensor Network Node

Quasimodo4

warehouses:sense &control

tracingproducts:

actievelabels

trains:seat

reservation

health care: on-body networks

Wireless Sensor Networks

Quasimodo5

Myrianed: a WSN with Gossiping

ant colonies

message exchange by gossiping:

local communication, global effect

inspired by biology: and also by girls:

cell interaction

Quasimodo6

• Communication inspiredon biology and human interaction

• Epidemic communication

• Analogy: spreading arumor or a virus

• MYRIANED “GOSSIP” protocol

• RF broadcast (2.4 Ghz ISM)


Quasimodo7







Quasimodo8







Quasimodo9







Quasimodo10







Quasimodo11







Quasimodo12







Quasimodo13







Quasimodo14







Quasimodo15







Quasimodo16

RF ANTENNA

RF TRANCEIVER

(NORDIC SEMI) CPU

(ATMEL XMEGA128)

I/O INTERFACES

Myrianed

1024 node experiment

Quasimodo17

WSN as an Integration Problem

Wireless Sensor Network

• integrating interacting

components = nodes

• many: > 1000

• identical

– but third-party nodes?

• nodes interact using gMAC protocol

(gossip Medium Access Control)

Quasimodo18

1. define interface

– gMAC protocol

(gossip Medium Access Control)

2. verify gMAC protocol

– using Timed Automata and UPPAAL

– defects found and repaired

3. test ‘each’ node in isolation

for compliance with the gMAC protocol

– also third-party nodes

WSN Integration: Model-Based

gMAC

gMAC

gMAC

test ‘each’ node in isolation

for compliance with the gMAC protocol

Quasimodo19

gMAC Behaviour

Node 2

Node 1

Node 3

Node 5

Node 4

total 32 bytes2 bytes slot nrMessage:

Quasimodo20

WSN: Model-Based Testing

Model-based testing of a single node:

• protocol conformance test

of the gMAC protocol

• according to ISO 9646

• local test method

• time is important in gMAC:

real-time model-based testing

Quasimodo21

Model-Based Testing

Quasimodo22

SUT

System Under Test

pass fail

Developments in Testing 1

1. Manual testing

Quasimodo23

SUT

pass fail

test execution

TTCNTTCNtestcases

1. Manual testing

2. Scripted testing


Quasimodo24

system

model

SUT

TTCNTTCNTestcases

pass fail

model-basedtest

generation

test execution

1. Manual testing

2. Scripted testing

3. Model-based

testing


Quasimodo25

Validation, Verification, and Testing

model properties

SUT

ideasideas

concrete

realizations

ideas

wishes

abstract

models,

math

validation

testing

verification

testing

validation

Quasimodo26

Un-Timed Model-Based Testing

with Labelled Transition Systems

Quasimodo27

Approaches to Model-Based Testing

Several modelling paradigms:

• Finite State Machine

• Pre/post-conditions

• Labelled Transition Systems

• Programs as Functions

• Abstract Data Type testing

• Timed Automata

• . . . . . . .

Labelled Transition Systems

Timed Automata

Quasimodo28

Models: Labelled Transition Systems

states

output actions

transitions

initial state

? = input

! = output

?coin

?button

!alarm ?button

!coffee

Labelled Transition System: ⟨⟨⟨⟨ S, LI, LU, T, s0 ⟩⟩⟩⟩

input actions

Quasimodo29

SUT passes tests

SUTconforms to

model

⇔⇔ ⇔⇔

system

model

SUT

TTCNTTCNTest

cases

pass fail

test execution

model-basedtest

generation

Model-Based Testing

SUTconforms to

model

Quasimodo30

MBT with Labelled Transition Systems

LTS

model

SUT

TTCNTTCNTest

cases

pass fail

LTStest

execution

iocotest

generation

input/output

conformance

ioco

set of

LTS tests

SUT passes tests

SUT ioco model

⇔⇔ ⇔⇔

Quasimodo31

p δδδδ p = ∀∀∀∀ !x ∈∈∈∈ LU ∪∪∪∪ {ττττ} . p !x

out ( P ) = { !x ∈ LU | p !x , p∈P } ∪∪∪∪ { δδδδ | p δδδδ p, p∈P }

Straces ( s ) = { σσσσ ∈ ( L ∪∪∪∪ {δδδδ} )* | s σσσσ }

p after σσσσ = { p’ | p σσσσ p’ }

Conformance: ioco

i ioco s =def ∀∀∀∀ σσσσ ∈∈∈∈ Straces (s) : out (i after σσσσ) ⊆⊆⊆⊆ out (s after σσσσ)

Quasimodo32

i ioco s =def ∀∀∀∀ σσσσ ∈∈∈∈ Straces (s) : out (i after σσσσ) ⊆⊆⊆⊆ out (s after σσσσ)

Intuition:

i ioco-conforms to s, iff

• if i produces output x after trace σ,

then s can produce x after σ

• if i cannot produce any output after trace σ,

then s cannot produce any output after σ ( quiescence δ )

Conformance: ioco

Quasimodo33

!coffee

?dime

?quart

?dime?quart

?dime?quart

?dime

!choc

?quart

!tea

!coffee

?dime

!tea

specificationmodel

Example: ioco

ioco

ioco

ioco

ioco

δδδδ

?dime

!coffee

?dime

!choc

?dime

!tea

Quasimodo34

Algorithm to generate a test case t(S)

from a transition system state set S, with S ≠ ∅ ( initially S = s0 after ε ).

Apply the following steps recursively, non-deterministically:

1 end test case

pass

2 supply input !a

!a

t ( S after ?a ≠ ∅∅∅∅ )

Test Generation Algorithm: ioco

allowed outputs (or δ): !x ∈out (S)

forbidden outputs (or δ): !y ∉ out (S )

3 observe all outputs

fail

t ( S after !x )

fail

allowed outputsforbidden outputs?y

θθθθ?x

fail

t ( S after !x )

fail

allowed outputsforbidden outputs

?y ?x

Quasimodo35

MBT with Labelled Transition Systems

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTStest

execution

iocotest

generation

input/output

conformance

ioco

set of

LTS tests

SUT passes tests

SUT ioco model

⇔⇔ ⇔⇔ exhaustivesound ⇑⇑⇑⇑ ⇓⇓⇓⇓

Quasimodo36

Real-Time Model-Based Testing

with Timed Automata

Quasimodo37

? money

? button1 ? button2

! coffee! tea

test case

fail

! money

! button1

?coffee

fail

? tea ϑϑϑϑ

pass

specification

Untimed MBT: ioco

Quasimodo38

? money

? button1 ? button2

test case

fail

! button1 @ 5 sec

?coffee

fail

? tea ϑϑϑϑ

pass

specification

Real-Time MBT

! money @ 5 sec

[ c ≥≥≥≥ 8 ] ->! tea

c := 0

c < 10 c < 15

[ c ≥≥≥≥ 5 ] ->! coffee

c := 0

sut ? msut rtioco s

39

Real-Time Model-Based Testing

� Approach: Extension of LTS: Timed LTS

Extension of ioco: real-time ioco

� Challenges:

♦ Is time input or output ?

♦ Quiescence: How long is there never eventually no output?

� Literature:

♦ Nielsen, Mikucionis, Skou, Larsen

♦ Krichen, Tripakis

♦ Brandán Briones, Brinksma

♦ Bohnenkamp, Belinfante

♦ Khoumsi, Jéron, Marchand

♦ Schmaltz, Tretmans

40

!coffee

?but1?but2

?but1 ?but2

?but1 ?but2

Untimed Model-Based Testing

!coffee

?but

!tea

spec

LTS : ⟨⟨⟨⟨ Q , LI , LU , T , q0 ⟩⟩⟩⟩

Observable actions: LI , LU

Specifications are LTS

Implementations are assumed

to behave as input-enabled LTS

Implementation relation: ioco

41

Timed Input-Output Transition Systems

c>=5

c:=0

c<10

?but

!coffee

TIOTS : ⟨⟨⟨⟨ Q, LI, LU, R≥0, T, q0 ⟩⟩⟩⟩

Observable actions: LI, LU

delay d ∈ R≥0

Specifications are Timed LTS

Implementations are assumed

to behave as input-enabled Timed LTS

42

i ioco s =def ∀σ ∈ traces (s) : out (i after σ) ⊆ out (s after σ)

Some Timed Implementation Relations

↓↓↓↓

?↓↓↓↓

?↓↓↓↓

?↓↓↓↓

tiocoX

43

i ioco s =def ∀σ ∈ traces (s) : out (i after σ) ⊆ out (s after σ)

A Timed Implementation Relation

↓↓↓↓outR

↓↓↓↓rtioco

ttraces ( s ) = { σ ∈ ( L ∪∪∪∪ R≥0 )* | sσ

}

p aftert σ = { p’ | p σ p’, σ ∈ ( L ∪∪∪∪ R≥0 )* }

δ ( p ) =

outR ( p ) = { x ∈ LU ∪∪∪∪ R≥0 | p x }

↓↓↓↓aftert

↓↓↓↓ttraces

44

rtioco rtioco

rtioco

rtioco

c>=5

c:=0

c<10

?but

!coffee

c==7

c:=0

c<=7

?but

!coffee

c>=7

c:=0

c<9

?but

!coffee

c>=3

c:=0

c<=9

?but

!coffee

c:=0

true

?butrtioco

45

Algorithm to generate a test case t(S)

from a timed transition system

Apply the following steps recursively, non-deterministically:

1 end test case

pass

2 supply input !a

!a@0

t ( S after ?a ≠ ∅∅∅∅ )

Test Generation Algorithm: rtioco

allowed outputs (or d): !x ∈out (S)

forbidden outputs (or d): !y ∉out (S )

3 observe all outputs

fail

t ( S after !x )

fail

allowed outputsforbidden outputs?y

d?x

fail

t ( S after !x )

fail

allowed outputsforbidden outputs

?y ?x

c=0

c<=d

Quasimodo46

Model-Based Testing Tools

Quasimodo47

• AETG

• Agatha

• Agedis

• Autolink

• Conformiq

Qtronic

• Cooper

• Uppaal-Cover

• G∀st

• Gotcha

• JTorX

• MaTeLo

• Phact/The Kit

• QuickCheck

• Reactis

• RT-Tester

• SaMsTaG

• Smartesting

Test Designer

• Spec Explorer

• Statemate

• STG

Some MBT Tools

• TestGen (Stirling)

• TestGen (INT)

• TestComposer

• TGV

• TorX

• TorXakis

• T-Vec

• Uppaal-Tron

• Tveda

• . . . . . .

Tron

TorXakis

JTorX

Tron

Quasimodo48

• On-the-fly test generation and test execution

• Implementation relation: (rt)ioco

• Mainly applicable to reactive systems / state based systems

Uppaal

TronSUT

observe

output

offerinput

nextinput

modelcheckoutput

passfailinconclusive

Tron, JTorX, TorXakis: On-the-Fly MBT

Quasimodo49

Back to the Challenge:

The Wireless Sensor Network Node

Quasimodo50


Model-based testing of a single node:

• protocol conformance test

of the gMAC protocol

• according to ISO 9646

• local test method

• time is important in gMAC:

real-time model-based testing

Quasimodo51

WSN Node in Protocol Layers

node 1

Application

Layer

gMAC

Layer

Radio

Layer

Application

Layer

gMAC

Layer

Radio

Layer

node 2

Application

Layer

gMAC

Layer

Radio

Layer

node 3

Medium Layer

application

interface

radio

interface

Upper Tester

Lower Tester

Quasimodo52

Local Test Method

application

interface

radio

interface

Upper Tester

Lower Tester

GMAC

layer

First approach:

• only software, on host

• simulated, discrete time

if(gMacFrame.currentSlotNumber >

gMacFrame.lastFrameSlot) {

gMacFrame.currentSlotNumber = 0;

gMacFrame.syncState = gMacNextFrame.syncState;

if (semaphore.appBusy == 1) {

mcTimerSet(SLOT_TIME);

mcPanic(MC_FAULT_APPLICATION);

return;

}

Hardware

Software

Quasimodo53

WSN: Test Architecture

Upper Tester

Lower Tester

GMAC layer:

Software

Model-Based

Test Tool:

• Uppaal-Tron

• JTorX

• TorXakis

sockets

Adapter

• transform

messages

• synchronize

simulated

time

sockets

Quasimodo54

SUT

TTCNTTCNTest

cases

model-basedtest

generation

test execution


system

model

model-based

test tool:

Uppaal-Tron

adapter

pass fail

test runs

WSN software on PC

(vague)

descriptionsguru

ad-hoc model

learning

rtioco

TA model

Quasimodo55

SUT

TTCNTTCNTest

cases

model-basedtest

generation

WSN: Test-Based Modeling

system

model

Uppaal-Tron

TorXakis

JTorX

adapter

pass fail

test runs

WSN software on PC

???Make a model

from observations

made during testing

Quasimodo56

WSN: A Timed Automaton Model

Quasimodo57

Demo

Quasimodo58

Concluding Remarks

Quasimodo59

Quasimodo Model-Based Testing Cases

• Chess: WSN gMAC protocol

• Hydac: model-based testing of existing controller for safety

• Terma: Modular Multi-Spectral Imaging Array

• NovoNordisk: off-line testing

• Dutch electronic biometric passport

• InterNLnet: Automated trust Anchor Updating in DNSSEC

Quasimodo

Quasimodo60

MBT Lessons

• Model construction

– leads to detection of design and specification errors

– how to get a valid model?

• Adapter/test environment

– specific for each system

– sometimes laborious, but not specific for MBT

• Longer and more flexible tests

– full automation : test generation + execution + analysis

– easy to repeat for regression testing, modifications, .....

Quasimodo61

MBT Perspectives

Model based formal testing can improve the testing process :

• model is precise and unambiguous basis for testing

– design errors found during validation of model

• longer, cheaper, more flexible, and provably correct tests

– easier test maintenance and regression testing

• automatic test generation and execution

– full automation : test generation + execution + analysis

• extra effort of modelling compensated by better tests

– is modelling really an extra effort?

• (commercial) model-based testing tools become available

Quasimodo62

Thank you !

model-based testing - aalborg universitet · model-based testing 3. un-timed model-based testing...

Documents