emmanuel fleury kim guldstrand larsen and jan tretmans

UCb

Emmanuel Fleury Kim Guldstrand Larsen

and

Jan Tretmans

Test & VerificationTest & Verification

2Kim G. Larsen

UCb

Research ProfileDistributed Systems & Semantics Unit

Semantic Models concurrency, mobility, objects real-time, hybrid systems

Validation & Verificationalgorithms & tools

Construction real-time & network systems

3Kim G. Larsen

UCb BRICS Machine Basic Research in Computer Science, 1993-2006

30+40+40 Millkr

100

100

Aalborg Aarhus

Tools

Other revelvant projects CISS, ARTIST, AMETIST

4Kim G. Larsen

UCb

Tools and BRICS

Logic• Temporal Logic• Modal Logic• MSOL • •

Algorithmic• (Timed) Automata Theory• Graph Theory• BDDs• Polyhedra Manipulation• •

Semantics• Concurrency Theory• Abstract Interpretation• Compositionality• Models for real-time & hybrid systems• •

HOL TLP

Applications

PVS ALF

SPINvisualSTATE UPPAAL

5Kim G. Larsen

UCb

A very complex system

Klaus Havelund, NASA

6Kim G. Larsen

UCb

Rotterdam Storm Surge Barrier

7Kim G. Larsen

UCb

Spectacular Bugs

ARIANE-5 INTEL Pentium II floating-point division

470 Mill US $ Baggage handling system, Denver

1.1 Mill US $/day for 9 months Mars Pathfinder Radiation theraphy, Therac-25 ……. More in JPK, CW

8Kim G. Larsen

UCb Embedded Systems

80% of all existing software is embedded in interacting devices.

Demand on increasing functionality with minimal resources.

9Kim G. Larsen

UCb

A simple program

Int x

Process INCdo:: x<200 --> x:=x+1od

Process DECdo:: x>0 --> x:=x-1od

Process RESETdo:: x=200 --> x:=0od

fork INC; fork DEC; fork RESET

Int x

Process INCdo:: x<200 --> x:=x+1od

Process DECdo:: x>0 --> x:=x-1od

Process RESETdo:: x=200 --> x:=0od

fork INC; fork DEC; fork RESET

Which values mayx take ?

Questions/Properties:E<>(x>100)E<>(x>200)A[](x<=200)E<>(x<0)A[](x>=0)Possibly

Always

10Kim G. Larsen

UCb

Introducing, Detecting and Repairing Errors Liggesmeyer 98

11Kim G. Larsen

UCb

Introducing, Detecting and Repairing Errors Liggesmeyer 98

12Kim G. Larsen

UCb

Suggested Solution?

Model based validation, verfication and testing

of software and hardware

13Kim G. Larsen

UCb

Verification & Validation

Design Model Specification

Analysis

Implementation

Testing

14Kim G. Larsen

UCb


Design Model SpecificationVerification & Refusal

AnalysisValidation

Implementation

Testing

UML

SDL

15Kim G. Larsen

UCb



AnalysisValidation

Implementation

Testing

UML

SDL

ModelExtraction

AutomaticCode generation

16Kim G. Larsen

UCb



AnalysisValidation

Implementation

Testing

UML

AutomaticCode generation

AutomaticTest generation

SDL

ModelExtraction

17Kim G. Larsen

UCb

How?

Unified Model = State Machine!

a

b

x

ya?

b?

x!

y!b?

Control states

Inputports

Outputports

18Kim G. Larsen

UCb

TamagotchiA C

Health=0 or Age=2.000

B

Passive Feeding Light

Clean

PlayDisciplineMedicine

Care

Tick

Health:=Health-1; Age:=Age+1

AA

A

A

AA

A

A

Meal

Snack

B

B

ALIVE

DEAD

Health:= Health-1

19Kim G. Larsen

UCb

SYNCmaster

20Kim G. Larsen

UCb

Digital Watch

21Kim G. Larsen

UCb

The SDL EditorThe SDL EditorThe SDL Editor

Process levelProcess level

22Kim G. Larsen

UCb S

PIN

, Gerald

Ho

lzman

n A

T&

T

23Kim G. Larsen

UCb visualSTATE

Hierarchical state systems

Flat state systems Multiple and inter-

related state machines

Supports UML notation

Device driver access

VVS w Baan Visualstate, DTU (CIT project)

24Kim G. Larsen

UCb ESTEREL

25Kim G. Larsen

UCb U

PP

AA

L

26Kim G. Larsen

UCb ‘State Explosion’ problem

a

cb

1 2

43

1,a 4,a

3,a 4,a

1,b 2,b

3,b 4,b

1,c 2,c

3,c 4,c

All combinations = exponential in no. of components

M1 M2

M1 x M2

Provably theoretical

intractable

27Kim G. Larsen

UCb

Train Simulator1421 machines11102 transitions2981 inputs2667 outputs3204 local statesDeclare state sp.: 10^476

BUGS ?

VVSvisualSTATE

Our techniuqes has reduced verific

ation

time w

ith several orders of magnitude

(ex 14 days to 6 sec)

28Kim G. Larsen

UCb

Modelling and Analysis

Software Model A

Requirement FYes, Prototypes Executable Code Test sequences

No!Debugging Information

Tools: UPPAAL, visualSTATE, ESTEREL, SPIN, Statemate, FormalCheck, VeriSoft, Java Pathfinder,…

TOOLTOOL

29Kim G. Larsen

UCb

Modelling and Analysis

Software Model A

Requirement FYes, Prototypes Executable Code Test sequences

No!Debugging Information

Tools: UPPAAL, visualSTATE, ESTEREL, SPIN, Statemate, FormalCheck, VeriSoft, Java Pathfinder,…

TOOLTOOL

BRICSBRICS

Semantics

Logic

Algorithmics

UCb

Finite State Machines

• Language versus behaviour• Determinism versus non-determinism• Composition and operations• Variants of state machines Moore, Mealy, IO automater, UML ….

Most fundamentae model in Computer Science: Kleene og Moore

31Kim G. Larsen

UCb

State Machines

Model of Computation• Set of states• A start state• An input-alfabet• A transition funktion, mapping input symbols and state to next state • One ore more accept states.• Computation starts from start state with a given input string (read from left to right)

inc

inc

inc

dec

dec

dec

Modulo 3 counter

inc inc dec inc inc dec inc

inc inc dec inc dec inc dec inc

input string

32Kim G. Larsen

UCb

State Machines

Variants

Machines may have actions/output associated withstate– Moore Machines.

01

2

inc

inc

inc

dec

dec

dec


0 1 2 1 2 0 2 1

inputstreng

outputstreng

33Kim G. Larsen

UCb

State Machines

Varianter

Machines may have actions/output associated with med transitions – Mealy Maskiner.

Transitions unconditional of af input (nul-transitions).

Several transitions for given for input and state (non-determinisme).

inc/0

inc/1

inc/2

dec/1

dec/0

dec/2


1 2 1 2 0 2 1

inputstreng

outputstreng

34Kim G. Larsen

UCb

State Machines

Variants

Symbols of alphabet patitioned in input- and output-actions (IO-automata)

inc?

inc?

inc?

dec?

dec?

dec?

0! 1!

2!

0! 0! 0! inc? inc? 2! 2! dec? 1!

interaction

35Kim G. Larsen

UCb

Bankbokskode

To open a bank boxthe code most contain at least 2

To open a bank boxthe code most end with

To open a bank box the code most end with a palindrome.g:. O

B

G

……..

?

To open a bank boxthe code most end with or with

36Kim G. Larsen

UCb

Fundamental Results

Every FSM may be determinized accepting the same language (potential explosion in size).

For each FSM there exist a language-equivalent minimal deterministic FSM.

FSM’s are closed under Å and [

FSM’s may be described as regular expressions (and vise versa)

37Kim G. Larsen

UCb

Composition

a

cb

1 2

43

1,a 4,a

3,a 4,a

1,b 2,b

3,b 4,b

1,c 2,c

3,c 4,c

All combinations=exponential in no of machines

All combinations=exponential in no of machines

M1 M2

M1 x M2

38Kim G. Larsen

UCb Composition FSM, Moore & Mealy

A

B

X

Y

I I

AX

BY

I

39Kim G. Larsen


A

B

X

Y

I I

AX

BY

I

O1 O2OO1+O2

40Kim G. Larsen


A

B

X

Y

I I

AX

BY

I

O1 O2OO1+O2

!Y,Z A Z

41Kim G. Larsen


A

B

X

Y

I I

AX

BY

O1 O2O

Y,Z A

42Kim G. Larsen

UCb Composition IO Automater (2-vejs synkronisering)

A

B

X

Y

h! h?

AX

BY

43Kim G. Larsen

UCb Composition IO Automater

A

B

X

Y

h! h?

AX

BY

C

k!

CX

k!

44Kim G. Larsen

UCb Mutual Exclusion

Token

45Kim G. Larsen

UCb Mutual Exclusion

Semafor

UCb

Automatisk Error Detection

• Reachability• Generic properties

47Kim G. Larsen

UCb

Udforskning af TilstandsrumErklæret tilstandsrum

Reachable

Start tilstand

a

cb

1 2

43

48Kim G. Larsen

UCb

Udforskning af tilstandrum

Erklæret tilstandsrumErklæret tilstandsrum

Forlæns iteration

start

Baglæns iteration

mål

49Kim G. Larsen

UCb Gensidig Udelukkelse

Token

50Kim G. Larsen

UCb Gensidig udelukkelse Forward Reachability

I1 I20

Token

51Kim G. Larsen


I1 I20

T1 I20 I1 T2

0

Token

52Kim G. Larsen


I1 I20

T1 I20

T1 T20

I1 T20

C1 I20

Token

53Kim G. Larsen


I1 I20

T1 I20

T1 T20

I1 T20

C1 I20

C1 T20

I1 I2T

Token

54Kim G. Larsen


I1 I20

T1 I20

T1 T20

I1 T20

C1 I20

C1 T20

I1 C2T

T1 T2T

T1 C2T

I1 T2T

T1 I2T

I1 I2T

Token

55Kim G. Larsen


I1 I2F

T1 I2F

T1 T2F

I1 T2F

I1 C2T

T1 C2T

Semafor

C1 I2T

C1 T2T

56Kim G. Larsen

UCb

Generiske egenskaber

Non-determinismeTilstande der ikke aktiveresTransitioner der ikke brugesInput der ikke processeresOutput der ikke genereresLokal deadlockSystem deadlock Kan alle reduceres til

REACHABILITYKan alle reduceres til

REACHABILITY

UCb

Automatic Test Generation

58Kim G. Larsen

UCb

Motivation

Testing = sample executions of system compared with requirements

Testing may identify errors but can not be used to exclude their presence.

Testing is the de-facto used method of validation

30-40% of the entire development process is concerned with testing.

59Kim G. Larsen

UCb

Black Box Testing

TESTER IMPLEMENTATION

input stimuli

output

conclusion

State Machine

State MachineState Machine

State Machine

60Kim G. Larsen

UCb

Black Box Testing


input stimuli

output

conclusion

State Machine


State Machine

closed/open TEST EXPECTED OUTPUTgogoobb closedgooobo openggggggggg closedooooggobo open……. ….

TEST EXPECTED OUTPUTgogoobb closedgooobo openggggggggg closedooooggobo open……. ….

61Kim G. Larsen

UCb

Black Box Testing


input stimuli

output

conclusion

State Machine


State Machine

closed/open TEST EXPECTED OUTPUTgogoobb closedgooobo openggggggggg closedooooggobo open……. ….

TEST EXPECTED OUTPUTgogoobb closedgooobo openggggggggg closedooooggobo open……. ….

MOORE’s Theorem:Hvis IMP antages at have mtilstande og SPEC har ntilstande da er det nok at testemht alle sekvenser af lgd n+m-1


62Kim G. Larsen

UCb

Black Box Testing


input stimuli

output

konklusion

Tilstandsmaskine

Tilstandsmaskine Tilstandsmaskine

Tilstandsmaskine

closed/open TEST EXPECTED OUTPUTggggobo open (closed) gggggoo closed (open)….. … ….. ………. ….

TEST EXPECTED OUTPUTggggobo open (closed) gggggoo closed (open)….. … ….. ………. ….



63Kim G. Larsen

UCb

Black Box Testing


input stimuli

output

konklusion

Tilstandsmaskine


Tilstandsmaskine

closed/open TEST EXPECTED OUTPUTggggobo open (closed) gggggoo closed (open)….. … ….. ………. ….

TEST EXPECTED OUTPUTggggobo open (closed) gggggoo closed (open)….. … ….. ………. ….



Problem:Antal test er ASTRONOMISK: k(n+m-1)

hvor k er antal symboler

64Kim G. Larsen

UCb

Black Box Testing


input stimuli

output

konklusion

Tilstandsmaskine


Tilstandsmaskine

closed/open

Control Flow CoverageEnhver transition skal fyresEnhver (lokal) tilstand skal nåsEnhver (ikke-triviel) guard skal kunne være både sand/falskDataflow Coverage …


Problem:Coverage kun afspecifikation –implementation behøver kun atvære dækket ganske lidt!


Løsning:Brug specifikation automatatil at (randomiseret) stimuleringog løbende check konsistens afimplementations reaktion

ToRX

65Kim G. Larsen

UCb

Black Box Testing


input stimuli

output

konklusion

Tilstandsmaskine


Tilstandsmaskine

closed/open





Løsning:Brug specifikation automatatil at (randomiseret) stimuleringog løbende check konsistens afimplementations reaktion

ToRX

UCb

VVS

Verification and Validation of Large Systems

DTU, Aalborg,Baan Visualstate

URLs://www.visualSTATE.com //www.it.dtu.dk/~jst/vvs/

67Kim G. Larsen

UCb BAAN VisualSTATE Tidligere BEOLOGIC

Beologic’s Products: salesPLUSsalesPLUS visualSTATEvisualSTATE

1980-95: Independent division of B&01995- : Independent company

B&O, 2M Invest, Danish Municipal Pension Ins. Fund

Customers:ABBB&ODaimler-BenzEricson DIAXESA/ESTECFORDGrundfosLEGOPBSSiemens ……. (approx. 90)

Verification Problems:• 1.000 components• 10400 states

Our techniques has reducedverification by an order of magnitude

(from 14 days to 6 sec)

•Embedded Systems•Simple Model•Verification of Std. Checks•Explicit Representation (STATEEXPLOSION)•Code Generation

68Kim G. Larsen

UCb visualSTATE 4.0 Product Modules

NavigatorPrototyper

Graphical Simulation of human interface panels

Presenter Prototyper for

distribution

Designer Diagram Designer Matrix Designer Text Editor

Tester Validator

SimulationAnimationAnalysis

VerificatorStatic verificationDynamic verification

Generator Coder Documentor

69Kim G. Larsen

UCb visualSTATE Prototyper

A virtual prototype ofa mobile telephone

GUI BuilderGUI ExecuterPick’n place of symbolsNo manual codingCustom designed

objects ActiveX controls Graphics libraries

70Kim G. Larsen

UCb visualSTATE Designer

Hierarchical state systems

Flat state systems Multiple and inter-

related state machines

Supports UML notation

Device driver access

71Kim G. Larsen

UCb

No local nor global dead-ends No never interpreted events No fired actions No conflicting transactions No unreachable states

All combinations are checked!

visualSTATE Tester Verification

100%Tested!

No bugs allowed!

72Kim G. Larsen

UCb

Train Simulator1421 machines11102 transitions2981 inputs2667 outputs3204 local statesDeclare state sp.: 10^476

BUGS ?

VVS

73Kim G. Larsen

UCb Experimental Breakthroughs

State Space St-of-Art ComBackSystem Mach.Declared Reach

Checks VisualST Sec MB Sec MB

VCR 7 10 5̂ 1279 50 <1 <1 6 <1 7JVC 8 10 4̂ 352 22 <1 <1 6 <1 6HI-FI 9 10 7̂ 1416384 120 1200 1.0 6 3.9 6Motor 12 10 7̂ 34560 123 32 <1 6 2,0AVS 12 10 7̂ 1438416 173 3780 6.7 6 5.7 6Video 13 10 8̂ 1219440 122 --- 1.1 6 1.5 6Car 20 10 1̂1 9.2 10 9̂ 83 --- 3.8 9 1.8 6N6 14 10 1̂0 6399552 443 --- 32.3 7 218 6N5 25 10 1̂2 5.0 10 1̂0 269 --- 56.2 7 9.1 6N4 23 10 1̂3 3.7 10 8̂ 132 --- 622 7 6.3 6Train1 373 10^136 --- 1335 --- --- --- 25.9 6Train2 1421 10^476 --- 4708 --- --- --- 739 11

Machine: 166 MHz Pentium PC with 32 MB RAM

---: Out of memory, or did not terminate after 3 hours.

VVS project BRICS/Aalborg, DTU, BAAN visualSTATE

74Kim G. Larsen

UCb Experimental BreakthroughsPatented

State Space St-of-Art ComBackSystem Mach.Declared Reach

Checks VisualST Sec MB Sec MB

VCR 7 10 5̂ 1279 50 <1 <1 6 <1 7JVC 8 10 4̂ 352 22 <1 <1 6 <1 6HI-FI 9 10 7̂ 1416384 120 1200 1.0 6 3.9 6Motor 12 10 7̂ 34560 123 32 <1 6 2,0AVS 12 10 7̂ 1438416 173 3780 6.7 6 5.7 6Video 13 10 8̂ 1219440 122 --- 1.1 6 1.5 6Car 20 10 1̂1 9.2 10 9̂ 83 --- 3.8 9 1.8 6N6 14 10 1̂0 6399552 443 --- 32.3 7 218 6N5 25 10 1̂2 5.0 10 1̂0 269 --- 56.2 7 9.1 6N4 23 10 1̂3 3.7 10 8̂ 132 --- 622 7 6.3 6Train1 373 10^136 --- 1335 --- --- --- 25.9 6Train2 1421 10^476 --- 4708 --- --- --- 739 11

Machine: 166 MHz Pentium PC with 32 MB RAM

---: Out of memory, or did not terminate after 3 hours.

Our techniques h

ave reduced

verification tim

e with

several orders of m

agnitude

(ex fro

m 14 days to 6 se

c)

UCb

Timed Automata =State Machines with Clocks

UPPAALA real time verification tool

76Kim G. Larsen

UCb

Hybrid & Real Time Systems

PlantContinuous

Controller ProgramDiscrete

Control Theory Computer Science

Eg.:Pump ControlAir BagsRobotsCruise ControlABSCD PlayersProduction Lines

Real Time SystemA system where correctness not only depends on the logical order of events but also on their timing

Real Time SystemA system where correctness not only depends on the logical order of events but also on their timing

sensors

actuators

TaskTask

TaskTask

77Kim G. Larsen

UCb Validation & VerificationConstruction of UPPAAL models

PlantContinuous

Controller ProgramDiscrete

sensors

actuators

TaskTask

TaskTask

a

cb

1 2

43

a

cb

1 2

43

1 2

43

1 2

43

a

cb

UPPAAL Model

Modelofenvironment(user-supplied)

Model oftasks(automatic)

78Kim G. Larsen

UCb

Intelligent Light Control

Off Light Brightpress? press?

press?

press?

WANT: if press is issued twice quickly then the light will get brighter; otherwise the light is turned off.

79Kim G. Larsen

UCb

Intelligent Light Control

Off Light Brightpress? press?

press?

press?

Solution: Add real-valued clock x

X:=0X<=3

X>3

80Kim G. Larsen

UCb

Timed Automata

n

m

a

Alur & Dill 1990

Clocks: x, y

x<=5 & y>3

x := 0

Guard Boolean combination of integer boundson clocks and clock-differences.

ResetAction perfomed on clocks

Transitions

( n , x=2.4 , y=3.1415 ) ( n , x=3.5 , y=4.2415 )

e(1.1)

( n , x=2.4 , y=3.1415 ) ( m , x=0 , y=3.1415 )

a

State ( location , x=v , y=u ) where v,u are in R

Actionused

for synchronization

81Kim G. Larsen

UCb

n

m

a

Clocks: x, y

x<=5 & y>3

x := 0

Transitions

( n , x=2.4 , y=3.1415 ) ( n , x=3.5 , y=4.2415 )

e(1.1)

( n , x=2.4 , y=3.1415 )

e(3.2)

x<=5

y<=10

LocationInvariants

g1g2 g3

g4

Timed Automata Invariants

Invariants ensure

progress!!

Invariants ensure

progress!!

82Kim G. Larsen

UCb

The Druzba MUTEX Problem

KimGerd

83Kim G. Larsen

UCb

The Druzba MUTEX Problem

84Kim G. Larsen

UCb

The Druzba MUTEX ProblemUsing the lightas semaphor

emmanuel fleury kim guldstrand larsen and jan tretmans

Documents