mts: controllable test objects
DESCRIPTION
Invited Talk, ISSTA 2nd International Workshop on End-to-end Test Script Engineering July 16, 2012, Minneapolis. Limitations of x-unit testing framework, MTS testing framework that combined test objects with procedural aspects of TTCN.TRANSCRIPT
MTS: Controllable Test Objects
Robert V. Binder 2nd International Workshop on End-to-end Test Script Engineering
July 16, 2012 Minneapolis
Copyright © 2012, Robert V. Binder
Overview
• Testing in Flatland
• Overview of MTS
• Lessons Learned
• Q&A
July 16, 2012 MTS - Controllable Test Objects 2
July 16, 2012 MTS - Controllable Test Objects 3
The xUnit Phenomena
jUnit
cppUnit
July 16, 2012 MTS - Controllable Test Objects 4
Incremental Testing Framework
TestObject
TestCaseTestSuite -Contains
0..1 0..*
0..1
-Contains
0..*
TestRunner TestLog+Contains
1 1
MyTestSuite
-Uses
1 1
MyTestObject
«uses»
July 16, 2012 MTS - Controllable Test Objects 5
Testing in Flatland
• Run A
p q r s t v
• Run B
r s t u
• Run C
t u
A p q B C p
q
r
s
t
u
B r s C
C t u
TestCases TestSuites
July 16, 2012 MTS - Controllable Test Objects 6
Straight Track
July 16, 2012 MTS - Controllable Test Objects 7
xUnit’s One Track Strategy
• Define and nest groups
• Run any sequence of groups
• No support for group or member
– Repeat
– Select
• Obstacle to any rich strategy
– Select, repeat
– Skip, Halt
– Time
– Broadcast
– Synchronize
July 16, 2012 MTS - Controllable Test Objects 8
Composition Isn’t Enough
• atom == test case
• ( ) group
• * quantification
• | select (or)
• How to do ( p | q )
• How to do (t ( p | q ))*
xUnit is like reg exp without * ? + |
A formal language without alternation and Kleene star
July 16, 2012 MTS - Controllable Test Objects 9
Composition Isn’t Enough Why Repeat?
• Iterate over collection
• Iterate over generation strategy
• Performance test over interval
• Capacity test to limits
Why Select?
• Respond to environment configuration/conditions
• Skip – Depends on earlier
– Don’t halt long complex test run
– Count failures for reliability estimate
July 16, 2012 MTS - Controllable Test Objects 10
Straight Track
July 16, 2012 MTS - Controllable Test Objects 11
Switching Tracks
July 16, 2012 MTS - Controllable Test Objects 12
MTS ARCHITECTURE AND FEATURES
July 16, 2012 MTS - Controllable Test Objects 13
MTS: Any App, Any Platform
Agent Host
Client Host Under Test
Agent HostConsole Host
MTS Console
MTS Test
Agent
MTS Test
Agent
Host Under Test may be
TEST RUN
REPORTS
MTS Remote
Agent
Client
Under Test
Server Host Under Test
MTS Remote
Agent
Server
Under Test
Cell Phone
PDA
Desktop
Server
Embedded Processor
Network Equipment
Access Point
Base Station
Client Host Under Test
MTS Remote
Agent
Client
Under Test
Server Host Under Test
MTS Remote
Agent
Server
Under Test
July 16, 2012 MTS - Controllable Test Objects 14
MTS Approach
• Test harness for distributed, end to end testing
– xUnit: composable test objects
– TTCN: adapters, concurrency, …
– Tcl: defacto standard for test script automation
July 16, 2012 MTS - Controllable Test Objects 15
About TTCN-3
• Procedural DSL similar to CORBA IDL
• Point of Observation and Control (adapters)
• Modular, but not necessarily composable
• Well-suited to model-based testing – Generate TTCN from MSCs,
FSMs, etc.
– Generate impl source from TTCN
– Compile & run impl
• Same test suite can be used on many implementations
July 16, 2012 MTS - Controllable Test Objects 16
Tcl
• Open source interpreter
• LISP-ish evaluation
iut.set {1} iut.set {$x} iut.set {[genX{$y}]} iut.set {[eval $codeForX]}
July 16, 2012 MTS - Controllable Test Objects 17
Tcl • Open source
interpreter
• LISP-ish evaluation
• Object-oriented
[incr Tcl]
July 16, 2012 MTS - Controllable Test Objects 18
MTS::TestObject
PPC 2002
YourTestObject
+getTestObjectClassName()
+getTestObjectHeadline()
+getTestObjectFileName()
+getTestObjectDir()
+getRemoteHostStatus()
+getTestRunRepeatCount()
+getTestRunIteration()
#require()
#setup()
#test()
#cleanup()
#ensure()
#check()
#run()
#submit()
#userLogEntry()
#userLogImage()
#evaluateCompareFrame()
#replayTestActionSequence()
+testObjectHeadline
MTS::TestObject
+require()
+setup()
+test()
+cleanup()
+ensure()
MyTestObject
MTS Framework
+connect()
+open()
+close()
+test()
+record()
Adapter
+getImage()
WM5
+getServiceWSDL()
+parseWSDL()
+queryForDictionary()
+queryForXML()
WebService
July 16, 2012 MTS - Controllable Test Objects 19
TestObject methods Skip rest of test object when its preconditions aren't met. Automatically called when a TestObject is run.
require
Create starting conditions for a test; allocate necessary resources. setup
Run tests. The user's implementation is expected to execute test actions leading to one or more test verdicts.
test
Restore pretest conditions; release resources the user test object has claimed.
cleanup
Skip or halt rest of a test run when postconditions aren't met. Automatically called after cleanup completes.
ensure
July 16, 2012 MTS - Controllable Test Objects 20
Using require ::itcl::class aTestCase{ inherit TestObject ::itcl::body aTestCase::require {} { if {[getRemoteHostStatus] eq "Connected"} { set remoteHostconnection true } else { userLogEntry "Remote Host not ready, skipping" set remoteHostconnection false } return $remoteHostconnection } ::itcl::body aTestCase::test {} { … }
July 16, 2012 MTS - Controllable Test Objects 21
Completion Actions return keyword
Current Test object Caller Test object Test objects run stack Counters
OK No effect, Destruct (ensure)
No effect. No effect. No change.
SKIP Destruct Continue with the next test object.
No effect. No change.
HALT Destruct Immediately exit the method that called this test object.
Terminate the test run. Percolate the HALT to the run stack.
No change.
EXCP Capture current Tcl errorInfo. Destruct the current test object.
Immediately exit the method that called this test object.
Terminate the test run. Percolate the EXCP to all test objects in the run stack.
Increment error counter.
July 16, 2012 MTS - Controllable Test Objects 22
Using ensure and Completion Actions ::itcl::class checkDir {inherit TestObject ::itcl::body checkDir::require {} { … } ::itcl::body checkDir::setup {} { … } ::itcl::body checkDir::test {} { … } ::itcl::body checkDir::cleanup {} { … } ::itcl::body checkDir::ensure { set currentDir [pwd] if { ! [regexp {mydir} $currentDir]} { return SKIP } else { return OK } }
July 16, 2012 MTS - Controllable Test Objects 23
MTS run command
run <target> repeat <times> <arg list>
• “Sources in” target file TestObject subclass
• Runs this TestObject
• Returns control to the caller
• Hardcoded target run myTestObject
• Target name in a variable run $anyTestObject
• Function computes target run [eval whichTestObject]
July 16, 2012 MTS - Controllable Test Objects 24
Selection and Repetition in MTS How to do ( p | q ) How to do (t ( p | q ))*
::itcl::body pq::test{} { if {[rhStatus]} { run remoteTest } else { run localTest } }
::itcl::body mySuite::test{reps} { run strategy1 -repeat $reps } ::itcl::body strategy1::test{} { run t run pq }
July 16, 2012 MTS - Controllable Test Objects 25
TestObject helpers
Start other test objects or create a test suite run <target>
Evaluate and log pass/fail conditions check <expression>
Time any action during a test run Timer <method>
Object interface for test run as a whole testRun <method>
Monitor remote host resource utilization rpm <method>
Check the remote host getRHostStatus myHost
Perform data-driven tests submit <args>
Annotate test run reports userLogEntry <args>
July 16, 2012 MTS - Controllable Test Objects 26
IMPLEMENTING MODAL CLASS TEST PATTERN
July 16, 2012 MTS - Controllable Test Objects 27
Modal Class Test Pattern α
ω
Two PlayerGam e ( )
~( ) ~ ( )
Gam e S tarte d
P la ye r 1
Served
P la ye r 2
Served
p2 _S tart ( ) /
s im u lateVolle y( )
p1 _W insVolle y( ) /
s im u lateVolle y( )
p1 _W insVolle y( )
[th is.p 1_ Score ( ) < 20 ] /
th is.p 1AddPoin t( )
s im u lateVolle y( )
p1 _S tart ( ) /
s im u lateVolle y( )
p2 _W insVolle y( )
[th is.p 2_ Score ( ) < 20 ] /
th is.p 2AddPoin t( )
s im u lateVolle y( )
p2 _W insVolle y( ) /
s im u lateVolle y( )p1 _W insVolle y( )
[th is.p 1_ Score ( ) == 20] /
th is.p 1AddPoin t( )
p2 _W insVolle y( )
[th is.p 2_ Score ( ) == 20] /
th is.p 1AddPoin t( )
p1 _IsW in ner( ) /
retu rn TR UE;
P la ye r 1
W o n
P la ye r 2
W o n p2 _IsW in ner( ) /
retu rn TR UE;
Ga m e S ta rted
P la yer 3
Serve d
p 3_ S tart ( ) /
s im ulateVo lley( )
p 1_ W insVo lle y( ) /
s im ulateVo lley( )
p 3_ W insVo lle y( )
[this.p 3_ Sco re ( ) < 2 0] /
th is.p3 Add Point ( )
s im ulateVo lley( )
p 2_ W insVo lle y( ) /
s im ulateVo lley( )
p 3_ W insVo lle y( )
[this.p 3_ Sco re ( ) == 2 0] /
th is.p3 Add Point ( )
P la yer 3
W on p 3_ IsW in ne r( ) /
return TR UE;
αTh ree P la yerGa m e ( ) / Two P la yerG am e ( )
p 3_ W insVo lle y( ) /
s im ulateVo lley( )
Tw oP layerG am e ( )
ω~( )
TwoPlayerGame
ThreePlayerGame
+TwoPlayerGame()+p1_Start( )+p1_WinsVolley( )-p1_AddPoint( )+p1_IsWinner( )+p1_IsServer( )+p1_Points( )+p2_Start( )+p2_WinsVolley( )-p2_AddPoint( )+p2_IsWinner( )+p2_IsServer( )+p2_Points( )+~( )
+ThreePlayerGame()+p3_Start( )+p3_WinsVolley( )-p3_AddPoint( )+p3_IsWinner( )+p3_IsServer( )+p3_Points( )+~( )
TwoPlayerGame
ThreePlayerGame
Game Started
Player 1Served
Player 2Served
p2_Start( ) /simulateVolley( )
p1_WinsVolley( ) /simulateVolley( )
p1_WinsVolley( )[this.p1_Score( ) < 20] /this.p1AddPoint( )simulateVolley( )
p1_Start( ) /simulateVolley( )
p2_WinsVolley( ) [this.p2_Score( ) < 20] /this.p2AddPoint( )simulateVolley( )
p2_WinsVolley( ) /simulateVolley( )
p1_WinsVolley( )[this.p1_Score( ) == 20] /this.p1AddPoint( )
p2_WinsVolley( )[this.p2_Score( ) == 20] /this.p1AddPoint( )
p1_IsWinner( ) /return TRUE; Player 1
WonPlayer 2Won
p2_IsWinner( ) /return TRUE;
α
ω
ThreePlayerGame( ) /TwoPlayerGame( )
~( )
Player 3Served
p3_WinsVolley( )[this.p3_Score( ) < 20] /this.p3AddPoint( )simulateVolley( )
p3_WinsVolley( )[this.p3_Score( ) == 20] /this.p3AddPoint( )
Player 3Won
p3_IsWinner( ) /return TRUE;
~( )
p3_Start( )/simulateVolley( )
p3_WinsVolley( ) /simulateVolley( )
p1_WinsVolley( ) /simulateVolley( )
p2_WinsVolley( ) /simulateVolley( )
p3_WinsVolley( ) /simulateVolley( )
~( )
July 16, 2012 MTS - Controllable Test Objects 28
Modal Class Test Design
Player 1 Served
Player 2 Served
Player 3 Won
omega
Player 3 Won
Player 3 Served
Player 3 Served
Player 1 Served
Player 2 Won
omega
Player 2 Won
Player 3 Served
Player 2 Served
Player 2 Served
Player 1 Served
Player 1 Won
omega
Player 1 Won
Player 3 Served
Player 2 Served
Player 1 Served
Gam eStartedalpha
3 p2_Start( )
6 p1_WinsVolley( )[this.p1_Score( ) < 20]
2 p1_Start( )
9 p2_WinsVolley( ) [this.p2_Score( ) < 20]
8 p2_WinsVolley( )
7 p1_WinsVolley( ) [this.p1_Score( ) == 20]
10 p2_WinsVolley( ) [this.p2_Score( ) == 20]
1 ThreePlayerGame( )
4 p3_Start( )
5 p1_WinsVolley( )
14 p1_IsWinner( )
15 p2_IsWinner( )
17 ~( )
12 p3_WinsVolley( ) [this.p3_Score( ) < 20]
13 p3_WinsVolley( ) [this.p3_Score( ) == 20]
16 p3_IsWinner( )
11 p3_WinsVolley( )
1
2
3
4
8
11
7
6
9
11
10
5
12
13
8
5
17
14
17
15
17
16
*
*
*
*
*
*
Sneak path tests not shown Note iterations
July 16, 2012 MTS - Controllable Test Objects 29
Implementing Modal Class Design xUnit
• TestCase for each transition (20) • Hand hack TestSuite for each
root to leaf path (17) and one to run them all – addTest – addTestSuite
• How to iterate transitions? • Embed strategy in single large test
object? – Very smelly
• Develop code generator that writes test suites?
MTS
• TestObject for each transition (20)
• Top level TestObject uses run to call transition test objects – Iteration trival
• Parameterized strategies
– N+
– Markov
– Etc.
• Parameterized state spec
July 16, 2012 MTS - Controllable Test Objects 30
Dude, where’s my object?
1 1
TestCase objects TestSuite objects
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
16
xUnit MTS
TestObject objects
Player 1 Served
Player 2 Served
Player 3 Won
omega
Player 3 Won
Player 3 Served
Player 3 Served
Player 1 Served
Player 2 Won
omega
Player 2 Won
Player 3 Served
Player 2 Served
Player 2 Served
Player 1 Served
Player 1 Won
omega
Player 1 Won
Player 3 Served
Player 2 Served
Player 1 Served
Gam eStartedalpha
3 p2_Start( )
6 p1_WinsVolley( )[this.p1_Score( ) < 20]
2 p1_Start( )
9 p2_WinsVolley( ) [this.p2_Score( ) < 20]
8 p2_WinsVolley( )
7 p1_WinsVolley( ) [this.p1_Score( ) == 20]
10 p2_WinsVolley( ) [this.p2_Score( ) == 20]
1 ThreePlayerGame( )
4 p3_Start( )
5 p1_WinsVolley( )
14 p1_IsWinner( )
15 p2_IsWinner( )
17 ~( )
12 p3_WinsVolley( ) [this.p3_Score( ) < 20]
13 p3_WinsVolley( ) [this.p3_Score( ) == 20]
16 p3_IsWinner( )
11 p3_WinsVolley( )
1
2
3
4
8
11
7
6
9
11
10
5
12
13
8
5
17
14
17
15
17
16
*
*
*
*
*
*
July 16, 2012 MTS - Controllable Test Objects 31
Controllable Test Objects
July 16, 2012 MTS - Controllable Test Objects 32
Design Goal TTCN xUnit MTS
Select, Iterate, any level Yes No Yes
Generate with MBT strategies Yes Yes Yes
Logical/Physical separation Yes No Yes
Platform agnostic/robust Yes ? Yes
Channel agnostic/robust Yes ? Yes
Distributed control Yes No Yes
Minimize Test Agent resources NA No Yes
Minimize brittleness Yes No Yes
Intuitive interaction No Yes Yes
Composable No Yes Yes
MTS hard problem queue • Protocol to drive remote target host IUT from test host
– Bi-directional, lightweight (BEEP ?) – Platform and stack agnostic – Target environment control (remote command line) – Distributed adapter RPC
• Forced reflection to generate adapters • Logical/physical mapping in generative models • Scalable asynchronous remote agents
– Control strategy – Provisioning – Protocol
July 16, 2012 MTS - Controllable Test Objects 33
Question Time
July 16, 2012 MTS - Controllable Test Objects 34
Bob Binder [email protected] www.robertvbinder.com