model-based testing of executable statecharts
TRANSCRIPT
Model-BasedTes,ngofExecutableStatecharts
TomMens,AlexandreDecanSo-wareEngineeringLab
UniversityofMons,Belgium
TomMens–SATTOSE2016–Bergen,Norway–July2016
Many“agile”developmenttechniquesprovidelightweightapproachestofacilitatechangeandincreasereliabilityofso-ware
• Qualityassessment(e.g.badsmellsandrefactoring)• Defensiveprogramming(e.g.designbycontract)• Test-drivendevelopment(e.g.unittesSngandbehavior-drivendevelopment)
• DynamicverificaSonofbehaviouralproperSes
Weproposetoraisethesetechniquestothelevelofexecutable(statechart)models
Agileanddefensivedevelopment
2
TomMens–SATTOSE2016–Bergen,Norway–July2016
Futurework(spoiler)
FacilitateevoluSonofbehaviouraldesignmodels– DetecSngmodelsmells– Modelrefactoring
• E.g.spliXngupacomplexstatechartintomulSplestatecharts– SemanScvariaSon
• DetecSngifstatechartiscompaSblewithalternaSvesemanScs– Variabilityanalysis
• Considerproductfamilies(e.g.differentmicrowavevariants)andanalysecommonaliSesandvariabiliSesintheirstatechartmodels
– DesignspaceexploraSon• AnalyseprosandconsofsyntacScallydifferent,butsemanScallysimilarstatecharts
3
TomMens–SATTOSE2016–Bergen,Norway–July2016
• AdvancedmodeltesSng(focusofthistalk)• Contract-drivenmodeling• Test-drivenmodeling(unittesSngandBDDforstatecharts)
• DynamicverificaSon(propertystatecharts)
• Futurework• Modelqualityassessment(modelsmells)• Modelqualityimprovement(modelrefactoring)• Modelchecking• Modelvariabilityanalysis• DesignspaceexploraSon• ModelcomposiSonandscalability• SemanScvariaSon
Agileanddefensivemodelling
4
TomMens–SATTOSE2016–Bergen,Norway–July2016
Runningexample
Microwaveoven<<component>>
Input
<<component>>Door
opened() closed()
<<component>>Control ler
-power : integer-timer : integer
<<component>>Lamp
switch_on() switch_off()
<<component>>Power
reset() inc() dec()
<<component>>Heating
set(power : integer) on() off()
<<component>>Turntable
start() stop()
<<component>>Timer
inc() dec() reset() tick()
<<component>>Display
clear() set(i : integer) set(s : string)
<<component>>Cooking
start() stop()
<<component>>WeightSensor
item_placed() item_removed()
beep(d : integer)
<<component>>Beeper
Visual Paradigm Standard Edition(University of Mons)
5
TomMens–SATTOSE2016–Bergen,Norway–July2016
Runningexample
Usecasename:CookFoodSummary:Userputsfoodinoven,andovencooksfood.Assump,ons:Ovenhasbeenconfiguredwithweightsensorandturntable.Precondi,ons:Ovenisclosedandempty.Postcondi,ons:Ovenhascookedthefood.Ovenisclosedandempty.Basiccourseofac,on:1.Useropensdoor.2.Userputsfoodinovenandclosesdoor.3.UsersetscookingSmeviacontrolpanel.4.Userpressesstartbubon.5.Magnetronindicatorlightswitcheson.Magnetronstartscookingfood.6.RemainingcookingSmeisdisplayedconSnuously.7.SystemnoSfiesuserwhencookingSmehaselapsed.Magnetronindicatorlightswitchesoff.8.Useropensdoor,removesfoodfromoven,andclosesdoor.9.Systemclearsdisplayandresetsdefaultvaluesforcooking.
TomMens–SATTOSE2016–Bergen,Norway–July2016
Runningexample
Usecasename:CookFoodAlternatecourses:1a:Userpressesstartbubonwhiledoorisopen.Systemdoesnotstartcooking.3a:Userpressesstartbubonwhilenofoodisintheoven.Systemdoesnotstartcooking.3b:UserpressesstartbubonwhilecookingSmeiszero.Systemdoesnotstartcooking.5a:Useropensdoorduringcooking.Magnetronstopsandindicatorlightturnsoff.Userremovesfood,closesdoorandpressesStop.Gotostep9.5b:Useropensdoorduringcooking.Magnetronstopsandindicatorlightturnsoff.UserclosesdoorandpressesStarttoresumecooking.Gotostep5.5c:UserpressesStopduringcooking.Magnetronstopsandindicatorlightturnsoff.UserpressesStarttoresumecooking.Gotostep5.
TomMens–SATTOSE2016–Bergen,Norway–July2016
So-ware-controlledsystemsaredifficulttodevelop
Controlso-warecanbeverycomplex– ConSnuousinteracSonbetweenso-wareandhardware
– ConSnuousinteracSonwithexternalworldandusers
– Mustrespectfunc8onalrequirements• OvenshouldcookfoodplacedinovenwithspecifiedpowerandduraSon
– Mustrespectnon-func8onalrequirements• Ovenshouldstopsendingmicrowavesifdoorsareopened
10
TomMens–SATTOSE2016–Bergen,Norway–July2016
• AddpreciseanddynamicallyverifiablespecificaSonstoexecutableso-warecomponents(e.g.,methods,funcSons,classes)
• BasedonBertrandMeyer’s“DesignbyContract”• Theso-warecompomentshouldrespectacontract,composedof– precondi8ons– postcondi8ons– invariants
Contract-drivendevelopment
11
TomMens–SATTOSE2016–Bergen,Norway–July2016
Example(takenfromwww.eiffel.com/values/design-by-contract/introducSon)
Contract-drivendevelopment
classDICTIONARY[ELEMENT]featureput(x:ELEMENT;key:STRING)isrequirecount<=capacitynotkey.emptyensurehas(x)item(key)=xcount=oldcount+1endinvariant0<=countcount<=capacityend
12
TomMens–SATTOSE2016–Bergen,Norway–July2016
Contract-drivenmodelling
Contractsformicrowavecontroller
contextcontrollerinv:notsent(heaSng_on)oracSve(cookingmode)inv:Smer>=0inv:0<power<=MAXPOWER
contextcookingmodepre:Smer>0inv:Smer>=0inv:power==power@prepost:received(door_opened)orSmer==0
contextreadyinv:Smer>0
TomMens–SATTOSE2016–Bergen,Norway–July2016
Tellingstories
Story(eventdoor_opened,eventitem_placed,eventdoor_closed,eventSmer_dec).tell(interpreter)
14
TomMens–SATTOSE2016–Bergen,Norway–July2016
Exampleoffailingcontract
InvariantErrorState:controllerAsserSon:Smer>=0ConfiguraSon:[controller,doorclosed,closedwithitem,programmode,notready]Step:eventSmer_decinternaltransiSononclosedwithitem
15
TomMens–SATTOSE2016–Bergen,Norway–July2016
SoluSontofailingcontract
AddguardstotheacSonsassociatedtotheeventsthatincrementanddecrementpowerandSmer
Smer_dec[Smer>0]/Smer-=1power_inc[power<MAXPOWER]/power+=1power_dec[power>1]/power-=1
16
TomMens–SATTOSE2016–Bergen,Norway–July2016
Test-drivendevelopment
testnegaSve_Smer:Story(door_opened,item_placed,door_closed,Smer_dec).tell(statechart)statechart.execute()assertEqual(State(controller).Smer,0)testno_hea8ng_when_door_is_not_closed:Story(door_opened,item_placed,Smer_inc,cooking_start).tell(statechart)statechart.execute()assertFalseacSve(cookingmode)assertFalsesent(heaSng_on)
testnegaSve_Smer...FAILtestno_heaSng_when_door_is_not_closed…ok===========================================AsserSonError:-1!=0----------------------------------------------------------------------Ran2testsin0.005sFAILED(failures=1)
WithoutguardsonSmer_decevent
17
TomMens–SATTOSE2016–Bergen,Norway–July2016
Test-drivendevelopment
testnegaSve_Smer:Story(door_opened,item_placed,door_closed,Smer_dec).tell(statechart)statechart.execute()assertEqual(State(controller).Smer,0)testno_hea8ng_when_door_is_not_closed:Story(door_opened,item_placed,Smer_inc,cooking_start).tell(statechart)statechart.execute()assertFalseacSve(cookingmode)assertFalsesent(heaSng_on)
18
testnegaSve_Smer...oktestno_heaSng_when_door_is_not_closed…ok----------------------------------------------------------------------Ran2testsin0.005sOK
WithguardsonSmer_decevent
TomMens–SATTOSE2016–Bergen,Norway–July2016
• IncludecustomertestpracScesintoTDD• EncouragecollaboraSonbetweendevelopers,QA,andnon-technicalstakeholders(domainexperts,projectmanagers,users)
• Useadomain-specific(non-technical)languagetospecifyhowthecodeshouldbehave– BydefiningfeaturespecificaSonsandscenarios
• Reducesthetechnicalgapbetweendevelopersandotherprojectstakeholders
Behaviour-DrivenDevelopment
19
TomMens–SATTOSE2016–Bergen,Norway–July2016
So-warebehaviourcanbedescribedinadomain-specific(non-technical)languagesuitedtonon-developers– usingtheGherkinlanguage– SupportedbyCucumberframeworkinmanylanguages
Behaviour-drivendevelopment
20
TomMens–SATTOSE2016–Bergen,Norway–July2016
Example(takenfromdocs.behat.org/en/v2.5/guides/1.gherkin.html)
Behaviour-drivendevelopment
Feature:ServecoffeeInordertoearnmoneycustomersshouldbeabletobuycoffeeScenario:BuylastcoffeeGiventhereis1coffeele-inthemachineAndIhavedeposited1dollarWhenIpressthecoffeebubonThenIshouldbeservedacoffee
21
TomMens–SATTOSE2016–Bergen,Norway–July2016
Behaviour-drivendevelopment
Feature:Nohea8ngifdoorisopenedScenario:Nohea8ngwhennothingisdoneGivenIdonothingAndIexecutethestatechartThenstatecooking_modeshouldnotbeacSveAndeventheaSng_onshouldnotbefiredScenario:Nohea8ngwhenitemisplacedGivenIsendeventdoor_openedWhenIsendeventitem_placedTheneventheaSng_onshouldnotbefiredScenario:Nohea8ngwhendoorisnotclosedGivenIsendeventdoor_openedAndIsendeventitem_placedWhenIsendeventdoor_closedTheneventheaSng_onshouldnotbefired
Firstvariant.
SSllreferstospecificdetailsofthestatechart(stateandeventnames)
22
1featurepassed,0failed,0skipped3scenariospassed,0failed,0skipped11stepspassed,0failed,0skipped,0undefinedTook0m0.005s
TomMens–SATTOSE2016–Bergen,Norway–July2016
Behaviour-drivendevelopment
Feature:Nohea8ngifdoorisopenedScenario:Nohea8ngwhennothingisdoneWhenIpowerupthemicrowaveThenheaSngshouldnotbeonScenario:Nohea8ngwhenitemisplacedGivenIopenthedoorWhenIplaceanitemThenheaSngshouldnotturnonScenario:Nohea8ngwhendoorisnotclosedGivenIopenthedoorAndIplaceanitemWhenIclosethedoorThenheaSngshouldnotturnon
Secondvariant.
Muchclosertonaturallanguage.Allstatecharts-specificconcepts
areabstractedaway.
23
TomMens–SATTOSE2016–Bergen,Norway–July2016
Coverageanalysis
Statecoverage:81.82%Enteredstates:controller(3)|doorclosed(4)|dooropened(2)|closedwithoutitem(3)|openedwithoutitem(2)|openedwithitem(2)|closedwithitem(1)|notready(1)|programmode(1)Remainingstates:cookingmode|readyTransi,oncoverage:16.67%Processedtransi,ons:openedwithoutitem[item_placed]->openedwithitem(2)closedwithoutitem[door_opened]->openedwithoutitem(2)openedwithitem[door_closed]->closedwithitem(1)
24
TomMens–SATTOSE2016–Bergen,Norway–July2016
Propertystatecharts
DefineandverifybehaviouralproperSesby1. instrumenSngthestatechartinterpreter2. intercepSngspecificacSonsofstatechartbeing
executed• entered(<NAMEOFSTATE>)• exited(<NAMEOFSTATE>)• consumed(<NAMEOFEVENT>)• sent(<NAMEOFEVENT>)• …
3. execuSngapropertystatechartthatverifiesadesirableorundesirableproperty
25
TomMens–SATTOSE2016–Bergen,Norway–July2016
Propertystatecharts
<<property statechart>>Heating does not start if door is opened
door is closed
door is opened
consumed(door_closed)
consumed(door_opened)
failure
sent(heating_on)
<<property statechart>>Heating must stop when door is opened
heating is off
heating is on
sent(heating_off)
sent(heating_on)
heating is on while door is opened
failure
consumed(door_opened)
consumed(tick)
sent(heating_off)
26
TomMens–SATTOSE2016–Bergen,Norway–July2016
Toolsupport
Sismic=SismicInteracSveStatechartModelInterpreterandChecker– PythonlibraryavailableonPythonPackageIndex(PyPI)
– releasedunderopensourcelicenceLGPLv3– Sourcecode• github.com/AlexandreDecan/sismic
– DocumentaSon• sismic.readthedocs.io
27
TomMens–SATTOSE2016–Bergen,Norway–July2016
Toolsupport
SismicsupportsallaforemenSonedconcepts– StatechartexecuSon– Designbycontract– UnittesSng– BDD– Coverageanalysis– Propertystatecharts– Andmore…
28
TomMens–SATTOSE2016–Bergen,Norway–July2016
Sismicfileformat
RepresenSngastatechartasaYAMLfile
rootstate:name:controllercontract:-always:notsent('heaSng_on')oracSve('cookingmode')-always:Smer>=0-always:0<power<=MAXPOWERini,al:doorclosedonentry:|power=DEFAULTSmer=0transi,ons:-event:input_cooking_stopacSon:|Smer=0
29
TomMens–SATTOSE2016–Bergen,Norway–July2016
Sismicfileformat
RepresenSngastatechartasaYAMLfile
states:-name:doorclosedini,al:closedwithoutitemstates:-name:closedwithoutitemtransi,ons:-event:door_openedtarget:openedwithoutitem-name:closedwithitemini,al:programmodeonexit:send('display_clear')transi,ons:-event:door_openedtarget:openedwithitem-event:input_Smer_incac,on:|Smer=Smer+1send('display_set',text='TIMER:%d'%Smer)…
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicExecuSngstatecharts
Stepwiseexecu8onofstatechartbehaviourfromsismic.ioimportimport_from_yaml
fromsismic.interpreterimportInterpreterfromsismic.modelimportEvent
withopen('microwave.yaml')asf:statechart=import_from_yaml(f)
interpreter=Interpreter(statechart)
interpreter.execute_once()MacroStep(None,[],>['controller','doorclosed','closedwithoutitem'],<[])
interpreter.queue(Event(’door_opened’))interpreter.execute_once()
MacroStep(Event(door_opened),[TransiSon(closedwithoutitem,openedwithoutitem,door_opened)],>['dooropened','openedwithoutitem'],<['closedwithoutitem','doorclosed'])
31
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicRunningstories
fromsismic.storiesimportStorystory=Story([Event('door_opened'),Event('item_placed'),Event('door_closed'),Event(’Smer_inc'),Event(’cooking_start'),Event(’Sck')])trace=story.tell(interpreter)
32
MacroStep(None,[],>['controller','doorclosed','closedwithoutitem'],<[]),MacroStep(Event(door_opened),[Transi8on(closedwithoutitem,openedwithoutitem,door_opened)],>['dooropened','openedwithoutitem'],<['closedwithoutitem','doorclosed']),MacroStep(InternalEvent(lamp_on),[],>[],<[]),MacroStep(Event(item_placed),[Transi8on(openedwithoutitem,openedwithitem,item_placed)],>['openedwithitem'],<['openedwithoutitem']),MacroStep(Event(door_closed),[Transi8on(openedwithitem,closedwithitem,door_closed)],>['doorclosed','closedwithitem','programmode','notready'],<['openedwithitem','dooropened']),…
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicRunningstories
MacroStep(InternalEvent(lamp_off),[],>[],<[]),MacroStep(Event(8mer_inc),[Transi8on(closedwithitem,None,8mer_inc)],>[],<[]),MacroStep(None,[Transi8on(notready,ready,None)],>['ready'],<['notready']),MacroStep(InternalEvent(display_set,text=TIMER:1),[],>[],<[]),MacroStep(Event(cooking_start),[Transi8on(ready,cookingmode,cooking_start)],>['cookingmode'],<['ready','programmode']),MacroStep(InternalEvent(hea8ng_set_power,power=900),[],>[],<[]),MacroStep(InternalEvent(hea8ng_on),[],>[],<[]),MacroStep(InternalEvent(lamp_on),[],>[],<[]),MacroStep(InternalEvent(turntable_start),[],>[],<[]),MacroStep(Event(8ck),[Transi8on(cookingmode,None,8ck)],>[],<[]),MacroStep(None,[Transi8on(cookingmode,programmode,None)],>['programmode','notready'],<['cookingmode']),MacroStep(InternalEvent(display_set,text=REMAINING:0),[],>[],<[]),MacroStep(InternalEvent(hea8ng_off),[],>[],<[]),MacroStep(InternalEvent(lamp_off),[],>[],<[]),MacroStep(InternalEvent(turntable_stop),[],>[],<[]),MacroStep(InternalEvent(beep,number=3),[],>[],<[]),MacroStep(InternalEvent(display_set,text=DONE),[],>[],<[])]
33
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicunittesSng
• Usingpython’sbuilt-inunibestmodule$python-munibestheaSng_unibest.py–v
deftest_no_heaSng_when_nothing_is_done(self):self.interpreter.execute()self.assertFalse(self.is_heaSng())deftest_no_heaSng_when_item_is_placed(self):events=map(Event,['door_opened','item_placed'])story=Story(events)story.tell(self.interpreter)self.interpreter.execute()self.assertFalse(self.is_heaSng())
34
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicBDD
• UsingPython’sbehavemodule$sismic-behavemicrowave.yaml--featuresheaSng.feature
frombehaveimportgiven,when,thenfromsismic.ioimportimport_from_yamlfromsismic.interpreterimportInterpreterfromsismic.interpreter.helpersimportlog_tracefromsismic.modelimportEvent@given('Iexecutethestatechart')defexecute_statechart(context):_execute_statechart(context,force_execuSon=True)@then('state{state_name}shouldbeacSve')defstate_is_acSve(context,state_name):assertstate_nameincontext._statechart.states,'Unknownstate{}'.format(state_name)assertstate_nameincontext._interpreter.configuraSon,'State{}isnotacSve'.format(state_name) 35
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicRegressiontesSng
Whenanerrorisencountered(e.g.duetofailingcontractorbug),story_from_tracecanreproducethescenariooftheobservedbehavior,whichcanbeusedasthebasisofaregressiontest.
36
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicCommunicaSngstatecharts• Statechartscancommunicatewithotherstatechartsorexternalcomponents(e.g.auserinterface)bysending/receivingevents
• Realisedbydynamicallybindingtheirstatechartinterpreters
37
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicCommunicaSngstatechartsExampleforsomeelevatorstatechart:Eventssentbybubonsarepropagatedtoelevatorelevator=Interpreter(import_from_yaml(open(‘elevator.yaml')))bubons=Interpreter(import_from_yaml(open(‘bubons.yaml')))bubons.bind(elevator)bubons.queue(Event(’floor_2_pushed'))bubons.execute()AwaiSngeventsinbubons:[Event(bubon_2_pushed)]AwaiSngeventsinbubons:[InternalEvent(floorSelected,floor=2)]AwaiSngeventsinelevator:[Event(floorSelected,floor=2)]elevator.execute()print('Currentfloor:',elevator.context.get('current'))Currentfloor:2
38
TomMens–SATTOSE2016–Bergen,Norway–July2016
SismicOtherfeatures
OthersemanScvariantsofstatecharts– outer-firstinsteadofinner-firstsemanScs;– changingpriorityofevents– …
DifferentwaysofdealingwithSme– RealSmeversussimulatedSme
39
TomMens–SATTOSE2016–Bergen,Norway–July2016
Conclusion
Wesupportvariouswaystoteststatechartmodels– Usingcontracts– Usingunittests– Usingdomain-specificfeaturesandscenarios(BDD)
– Usingpropertystatecharts
ImplementedinSismic,anopensourcePythonlibraryforinterpreSngstatecharts
40
TomMens–SATTOSE2016–Bergen,Norway–July2016
Futurework
• MoreadvancedtesSngtechniques– AutomaScgeneraSonofcontractsbasedonscenariospecificaSons
– AutomaScgeneraSonoftestsbasedoncontractspecificaSons
– MutaSontesSng– SupportforconSnuousintegraSon
• Explore/comparewith(dynamic?)modelcheckingtechniques– Basedontemporallogics,labeledtransiSonsystems,…– UsingDwyer’sspecificaSonpaberns
• Andmanymore…41
TomMens–SATTOSE2016–Bergen,Norway–July2016
Futurework
FacilitatestatechartevoluSon– DetecSngmodelsmells– Modelrefactoring
E.g.spliXngupacomplexstatechartintomulSplestatecharts– SemanScvariaSon
DetecSngifstatechartiscompaSblewithalternaSvesemanScs– Variabilityanalysis
Considerproductfamilies(e.g.differentmicrowavevariants)andanalysecommonaliSesandvariabiliSesintheirstatechartmodels
– DesignspaceexploraSonAnalyseprosandconsofsyntacScallydifferent,butsemanScallysimilarstatecharts
42