department of cis university of pennsylvania 1/31/2001 specification-based protocol testing hyoung...
Post on 21-Dec-2015
215 views
TRANSCRIPT
1/31/2001
Department of CISUniversity of Pennsylvania
Specification-based Protocol Testing
Hyoung Seok HongOleg Sokolsky
CSE 642
1/31/2001
Department of CISUniversity of Pennsylvania
Outline
• Introduction– Specifications, implementations– Conformance relations– Tests, testing architectures– Assumptions, etc.
• Formal protocol specifications• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing• Test generation for StateCharts
1/31/2001
Department of CISUniversity of Pennsylvania
Testing problem
• A specification gives a description of the system behavior.
• An implementation should behave according to the specification.
• Is this really true?
c
a
b
a
b
a
b
1/31/2001
Department of CISUniversity of Pennsylvania
Black-box vs. White-box testing
• Black-box: unknown system structure– Testing against reference specification– Use the specification interface for
• test selection, coverage criteria, result analysis
• White-box: known system structure– Testing against reference specification– Use system structure for
• test selection, etc.
1/31/2001
Department of CISUniversity of Pennsylvania
Protocol testing architectures
TesterLocal
architecture
IUT
• IUT – implementation under test
PCO
• PCO – point of control and observation
1/31/2001
Department of CISUniversity of Pennsylvania
Protocol testing architectures
UpperTester
IUT
PCO1
PCO2
underlying service layer
LowerTester
DistributedarchitectureCoordinatedarchitecture
Remotearchitecture
1/31/2001
Department of CISUniversity of Pennsylvania
General testing architecture
IUT
PCO
PCO PCO
Tester
Tester Tester
1/31/2001
Department of CISUniversity of Pennsylvania
Tests and test suites
• A test case defines:– A finite sequence of inputs – A finite sequence of expected outputs
• A test suite is a collection of tests to achieve a certain test coverage
• Testing process– test generation: construct a test suite– test application: execute each test– result evaluation: interpret outputs
1/31/2001
Department of CISUniversity of Pennsylvania
Test structure
• Test purpose– e.g., identify a state of the
implementation
• Test preamble– lead the IUT into known state
• Test body– invoke behavior corresponding to the test
purpose
• Checking of test results
1/31/2001
Department of CISUniversity of Pennsylvania
Modeling approach• Testing criteria on a concrete system
are hard to formulate
implementationdetailed
specification?
conformancerelation
• Abstract models make reasoning easier
abstract modelof specification
abstract modelof implementation
assumptions andtest hypothesis
assumptions andtest hypothesis
1/31/2001
Department of CISUniversity of Pennsylvania
Outline
• Introduction• Formal protocol specifications
– Formal specification languages
• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing• Test generation for StateCharts
1/31/2001
Department of CISUniversity of Pennsylvania
Formal specification languages
• Extended FSM– SDL– Estelle– StateCharts
• Process Algebras– LOTOS
• Algebraic Specification– Z, VDM
1/31/2001
Department of CISUniversity of Pennsylvania
Outline
• Introduction• Formal protocol specifications• Finite-state machine (FSM) testing
– FSM model– FSM fault models– FSM test selection
• Extended FSM (EFSM) testing• Test generation for StateCharts
1/31/2001
Department of CISUniversity of Pennsylvania
FSM model• A set of states• An initial state• Finite sets of input (X) and output (Y) events• Transfer function
x1
x2x3
• Output function/y1
/y2/y3
1/31/2001
Department of CISUniversity of Pennsylvania
FSM-based fault model
• Output faults• Transfer faults• Transfer faults with additional states• Additional or missing transitions• Control and data flow faults?
1/31/2001
Department of CISUniversity of Pennsylvania
Testing based on FSM models
• Testing methods based on control flow– Transition-tour (TT) method
• a single sequence of inputs to traverse all transitions
• simple but weak
– Unique-input-output (UIO) method• a test to identify each state in the specification
– Distinguishing sequence (DS) method• identify each state by the same test body• provides full coverage, but may not have a DS
1/31/2001
Department of CISUniversity of Pennsylvania
UIO testing method• Assumption: for each state, there is an input sequence that produces a unique output• Each test identifies one state
• s1 b/1• s2 a/0 a/0 b/1• s3 b/1 a/1
preamble body
s1 s2
s3
b/1
b/0
b/0
a/1
a/0a/0
1/31/2001
Department of CISUniversity of Pennsylvania
Outline
• Introduction• Formal protocol specifications• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing
– EFSM model– EFSM fault models– EFSM test selection
• Test generation for StateCharts
1/31/2001
Department of CISUniversity of Pennsylvania
EFSM model• A set of states• An initial state• Finite sets of input (X) and output (Y) events
• Transition relationx1
x2
x3
• A finite set of variables (V)
– Guardsb1
b2
b3
{output y1}
{update v2}
{output y3}– Update blocks
1/31/2001
Department of CISUniversity of Pennsylvania
Tests for EFSM models
• EFSM executions depend on input signals and data
• A test consists of:– test sequence– test data
• Executability problem:– Find data to execute the test sequence
• Undecidable, in general
1/31/2001
Department of CISUniversity of Pennsylvania
• Executability problem introduces an additional step in the testing process:– test generation: construct a test suite– test application: execute each test– result evaluation: interpret outputs
Test validation
– test validation: keep only executable tests– test application: execute each test
– result evaluation: interpret outputs
1/31/2001
Department of CISUniversity of Pennsylvania
Control and data flow testing
• EFSM executions are data-dependent• Control flow FSM testing methods are
not adequate for EFSM models• Data flow testing methods account for
data dependencies
1/31/2001
Department of CISUniversity of Pennsylvania
Data flow: definitions and uses
• Data variables are – defined by inputs and updates (def)– used in
• updates (c-use)• guards (p-use)
• Data-flow graph captures data dependencies
1/31/2001
Department of CISUniversity of Pennsylvania
Data flow graph
• Directed graph with nodes labeled with definitions and uses of variables
v=0{input u}
u<0{v:=u+1}
p-use vdef u
p-use uc-use udef v
1/31/2001
Department of CISUniversity of Pennsylvania
Data-flow coverage criteria
• all-def– test suite covers each definition at least
once
• all-use– cover each def-use association at least
once
• all-du-paths– exercise all paths from each definition of
a variable to every its use.
1/31/2001
Department of CISUniversity of Pennsylvania
all-use coverage
• Find at least one definition-free path for every def-use association
def v
use v
no definitons
of v
1/31/2001
Department of CISUniversity of Pennsylvania
Outline
• Introduction• Formal protocol specifications• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing• Test generation for Statecharts
1/31/2001
Department of CISUniversity of Pennsylvania
• EFSMs (FSMs + variables) + concurrency + hierarchy + communication + real-time
• Widely used for specifying real-time embedded HW/SW controllers
• Also used in most of object-oriented methodologies, e.g., UML
Statecharts
1/31/2001
Department of CISUniversity of Pennsylvania
A coffee vending machine
Statecharts = off
power-on/light-on; m:=0
power-off/light-off
Hierarchy +
notempty
emptyinc/m:=0
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
EFSMs +
busyidle
coffee
on
done/stop
Concurrency +
coffee[m>0]/start
Real-timetm(coffee,[3,5])/stop
Communication + coffee[m>0]/start;dec
1/31/2001
Department of CISUniversity of Pennsylvania
Test sequences for Statecharts
• The main purpose– What should implementations do?
•{power-on}/{light-on}, {inc}/{}, {coffee}/{start}
– What should not implementations do?•{power-on}/{light-on}, {coffee}/{}
• The main issue– How can we generate a finite and
reasonable number of test sequences?
1/31/2001
Department of CISUniversity of Pennsylvania
State coverage
off
power-on/light-on; m:=0
power-off/light-off
notempty
emptyinc/m:=0
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
busyidle
coffee
on
done/stop
coffee[m>0]/start
tm(coffee,[3,5])/stop
coffee[m>0]/start;dec
1/31/2001
Department of CISUniversity of Pennsylvania
Transition coverage
off
power-on/light-on; m:=0
power-off/light-off
notempty
emptyinc/m:=0
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
busyidle
coffee
on
done/stop
coffee[m>0]/start
tm(coffee,[3,5])/stop
coffee[m>0]/start;dec
1/31/2001
Department of CISUniversity of Pennsylvania
Data flow oriented coverage
off
power-on/light-on; m:=0
power-off/light-off
notempty
emptyinc/m:=0
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
busyidle
coffee
on
done/stop
coffee[m>0]/start
tm(coffee,[3,5])/stop
coffee[m>0]/start;dec
1/31/2001
Department of CISUniversity of Pennsylvania
Test generation from Statecharts (I)
• Basic idea– Transforms Statecharts into EFSMs
• Advantage– Can reuse the existing methods and
tools for EFSMs– Can handle infinite state space
• Disadvantage– Cannot determine the executability of
test sequences
1/31/2001
Department of CISUniversity of Pennsylvania
Test generation from Statecharts (II)
• Basic Idea– Test generation can be reduced into model
checking– TG: for each state s, is there a path leading to s– MC: a temporal logic formula EFs express this
property
• Advantage– Fully automatic– Generates only executable test sequences
• Disadvantages– Can handle only finite state space
1/31/2001
Department of CISUniversity of Pennsylvania
Another perspective
• Testing classes in object-oriented programs– Can we show an CLASS implementation
conforms to a Statechart specification?– Variables of Statecharts: data members of
classes– Events of Statecharts: member functions of
classes
• Testing inheritance in object-oriented programs– Can we reuse the test sequences of a super-
class for testing its subclasses ?
1/31/2001
Department of CISUniversity of Pennsylvania
A coffee-cocoa vending machine
off
power-on/light-on; m:=0
power-off/light-off
busyidle
cocoa
on
notemptyempty
inc/m:=0
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
tm(coffee,[3,5])/stop
cocoa[m>0]/start;dec
busyidle
coffee
tm(coffee,[3,5])/stop
coffee[m>0]/start;dec
VendingMachine
CoffeeVendingMachine
CocoaVendingMachine
Coffee-Cocoa
VendingMachine