1.Model-based Testing: Taking BDD/ATDD to the Next Level Robert V. Binder System Verification Associates rvbinder@sysverif.com #MoreModelsLessTestsChicago Quality Assurance Association February 25, 2014

2. The Incredible Vastness of TestingModelBased TestingEaten Alive by the Testing BackblobTell Me A StoryMore Models, More Tests Less Models, Less TestsModel Driven Development: Taking BDD/ATDD to the Next Level1 3. The Incredible Vastness of Testing 4. Saved by Revealing Sub-domains Grains of sand to fill the Grand Canyon ~ 1.68 1025 ~ 10 septillion Trivial Program Data space: 21024 Sequences: 21024 Execution space: 22048 1.34 10616Large subsets of any programs execution space are revealing: if a bug exists in any member of that space and it is executed, an observable failure results.But, we rarely know exactly which sub-domains are revealing until we test. Model Driven Development: Taking BDD/ATDD to the Next Level3 5. Its release time. Do you know where your bugs are? Aerospace MedicalIT0.7 0.8OtherAgile1.9 1.8 4.45.06.0 6.0Mobile Best In ClassAverage bugs per thousand equivalent source code lines, post-release4.7 Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .Model Driven Development: Taking BDD/ATDD to the Next Level4 6. Its release time. Do you know where your bugs are?Aerospace Medical ITBest In ClassOther325 375AgilePredicted total latent bugs, 500 KLOC Java925 875 2511 22022350 Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FPModel Driven Development: Taking BDD/ATDD to the Next Level5 7. Driven to Distraction Test Driven Development TDD Behavior Driven Development BDD Acceptance Test Driven Development What is it? How does it work? What can it do for me? ATDD Model-based Testing MBT Model Driven Development: Taking BDD/ATDD to the Next Level6 8. Eaten Alive by the Testing Backblob EATEN ALIVE BY THE TESTING BACKBLOB 9. Where does the time go? Total Number of Test CasesSprint 1Sprint 2Sprint 3Sprint 4}Tests/testing for new features Existing manual Test Cases not executedAvailable Dev/Test TimeExisting automated Test Cases not maintained Model Driven Development: Taking BDD/ATDD to the Next Level8 10. The Attack of the BackBlobhttp://www.youtube.com/watch?v=GODDLgM1gKo 11. Tell Me a StoryTELL ME A STORY 12. The Chat Server Logon List active users Post a message Broadcast message Logoff!!! Client EndpointClient EndpointModel Driven Development: Taking BDD/ATDD to the Next LevelChat Server 11 13. TDD BDD ATDD Test Driven Development Behavior Driven Development Acceptance Test Driven Development Programming strategies Test suites are scaffolding Test each method/feature at least once Rerun after any add/change Not testing strategiesModel Driven Development: Taking BDD/ATDD to the Next Level12 14. The TDD Red-Green Cycle Imagine a class and its responsibilities Code its API onlyLogon LogonGetStatus GetStatusLogoff LogoffModel Driven Development: Taking BDD/ATDD to the Next Level13 15. The Red-Green Cycle Imagine a class and its responsibilities Code its API onlyTestLogon Write and run a simple test it should failLogonGetStatusLogoffModel Driven Development: Taking BDD/ATDD to the Next Level14 16. The Red-Green Cycle Imagine a class and its responsibilities Code its API onlyTestLogon Write and run a simple test it should failLogonGetStatus Code implementation Rerun all tests expect passModel Driven Development: Taking BDD/ATDD to the Next LevelLogoff15 17. The Red-Green Cycle Imagine a class and its responsibilities Code its API only Write and run a simple test it should fail TestLogon Logon TestGetStatusGetStatus Code implementation Rerun all tests expect passLogoff Repeat until feature complete Model Driven Development: Taking BDD/ATDD to the Next Level16 18. The Red-Green Cycle Imagine a class and its responsibilities Code its API only Write and run a simple test it should failTestLogon Logon TestGetStatusGetStatus Code implementation Rerun all tests expect pass TestLogoffLogoff Repeat until feature complete Model Driven Development: Taking BDD/ATDD to the Next Level17 19. TDD Limitations Process, ActivitiesComponents, Test Objects No explicit linkage for user/customer needs, requirements, or stories Gaps UI automation Feature testing Model Driven Development: Taking BDD/ATDD to the Next Level18 20. Behavior Driven Development Process, ActivitiesComponents, Test Objects Features, Scenarios Feature Files Scenario A*DD Tool Scenario B Scenario CTest Objects *Unit Components Model Driven Development: Taking BDD/ATDD to the Next Level19 21. Acceptance Test Driven Development Process, ActivitiesFeatures, ScenariosUI Test Cases Feature Files Scenario A*DD Tool Scenario B Scenario CTest Scripts Test Objects Test Tooling *Unit System Under Test Model Driven Development: Taking BDD/ATDD to the Next Level20 22. Features and Scenarios Feature: story Scenario: instance of a story Steps: Given: pre-condition When: trigger Then: verification conditionFeature: Logon In order to use the chat server, As a validated chat client, I want to submit my credentials and establish a session Scenario: Submit Good Given A connection to When I send the Login Then I should see theModel Driven Development: Taking BDD/ATDD to the Next LevelCredentials the server is open message "Hi Chatter" message21 23. Overview Feature Files Scenario ASpecFlow Scenario B Scenario CTest Objects Nunit Visual Studio Componentshttp://www.specflow.org Enter features, scenarios Generates skeleton test code for each step Extracts step names Can run immediately Developer adds step code Test run reports CI integration, moreModel Driven Development: Taking BDD/ATDD to the Next Level22 24. 1. Create a Feature File SpecFlow generates a stub file Edit as necessary One feature 1:N scenariosModel Driven Development: Taking BDD/ATDD to the Next Level23 25. 2. Enter Scenario Steps Given Precondition Data State When Trigger API call, GUI action Then Verification conditions (expected results) Model Driven Development: Taking BDD/ATDD to the Next Level24 26. 3. Inspect Skeleton Test Code SpecFlow parses feature file Generates skeleton test code for each scenario and stepModel Driven Development: Taking BDD/ATDD to the Next Level25 27. 3. Add Test Case Code User adds more code as needed to implement each test caseModel Driven Development: Taking BDD/ATDD to the Next Level26 28. 4. Run Test Suite The generated test code can be run immediately Supports the redgreen cycleModel Driven Development: Taking BDD/ATDD to the Next Level27 29. SpecFlow BDD for Visual Studio Link Feature Files to test code Generates partial test codeFeature File contains statements in Gherkin Keywords + natural language Generates test code skeleton from stepsFeature File1:N Steps per ScenarioFeature1:N Scenarios per Feature FileSteps may be parameterizedStepsRowsVariables and variable bindings in RowsScenariosTest code generation using regular expression patternsIntegration with TFS, CI, Model Driven Development: Taking BDD/ATDD to the Next Level28 30. Modelbased Testing THE VERY MODEL OF A MODERN 31. What is a Model? Focus Smaller, simpler In IT systems Behavior Structure Content Model Driven Development: Taking BDD/ATDD to the Next Level30 32. IT Models Process, ActivitiesBehavior, Structure, ContentModel Driven Development: Taking BDD/ATDD to the Next LevelTechnology31 33. IT Models Notional (design) As built (documentation) Test Models Similar to notional and as-built Unique aspects 34. Test Models Focus on aspects that support testing goals Behavior Sequence and time Transformation Input, output function Verification (oracle) Coverage Risk Management Model Driven Development: Taking BDD/ATDD to the Next Level33 35. Chat Server LogonChat Server Must be first Cant be repeatedLogon GetStatus Logoff GetStatus Only after logon Returns number of calls since logon Logoff Only after logon Typical RFC status codes Model Driven Development: Taking BDD/ATDD to the Next Level34 36. Simple Test Model Logoffnc = 0 GetStatusSystem Under Test (SUT)LogonLogonLogonGetStatusassert (200)assert (401)assert (401)assert (401)Logoffnc++ GetStatusLogoff assert (200)Logoff assert (200) Model Driven Development: Taking BDD/ATDD to the Next Levelassert (200) assert (n == nc) 35 37. Chat Server Message Sequence Login a 1234 Hi Chatter a Login b secret Hi Chatter b List Users Users: a, b !!! Client EndpointClient EndpointModel Driven Development: Taking BDD/ATDD to the Next LevelChat Server36 38. Chat Server Message Sequence List Users Users: a, b Post Pumpkin Latte #mypics Ack a1 Pumpkin Latte #mypicsBcast a1 Pumpkin Latte #mypics Logoff b Ack Logoff b!!! Client EndpointClient EndpointModel Driven Development: Taking BDD/ATDD to the Next LevelChat Server37 39. Spec Explorer Demo Model-based testing tool Microsoft Research Used to test all MSFT APIs Robust and stable Visual Studio power tool C# code, not cartoons Generates standalone executable test suite Model Driven Development: Taking BDD/ATDD to the Next Level38 40. 1. Create Test Model Actions [Rule] tags a method for exploration Requires defines a exploration precondition must be true to explore Model Driven Development: Taking BDD/ATDD to the Next Level39 41. 2. Define Accepting State(s) Tag condition(s) that terminate an explorationModel Driven Development: Taking BDD/ATDD to the Next Level40 42. 3. Create Cord File Set options Define machines Limits Actions used in an exploration Similar to regular expression Composable Very powerful! Model Driven Development: Taking BDD/ATDD to the Next Level41 43. 4. Explore the Model Select any explorable machine Explore Generate test suite Run on-the-fly testModel Driven Development: Taking BDD/ATDD to the Next Level42 44. 5. Review Exploration Graph Shows action/event sequenceEllipse: model state computed by model program machineDiamond: observation event get something from SUT (input to the model program)Right click to inspect any itemModel Driven Development: Taking BDD/ATDD to the Next Level43 45. 6. Define machine(s) to generate test suites Set options to generate test code Use any machine or composite machineModel Driven Development: Taking BDD/ATDD to the Next Level44 46. 7. Explore the Test Suite machine Each root to leaf path becomes a test suiteModel Driven Development: Taking BDD/ATDD to the Next Level45 47. 8. Generate Test Code Explores the test machine Writes out test code using VS testing frameworkModel Driven Development: Taking BDD/ATDD to the Next Level46 48. 8. Generate Test Code Not intended to be maintained Re-generate any time Add helper code in the model to extend/customize Model Driven Development: Taking BDD/ATDD to the Next Level47 49. 9. Run the Test Suite Uses built-in Visual Studio test runnerAdapters Start/stop chat clients and chat server Other SUT setup/cleanupModel Driven Development: Taking BDD/ATDD to the Next LevelTest code sends messages, checks replies 48 50. Model State (test case inputs)10. Review Test Results Detailed run log Event Handler (pass/fail) Requirement TraceVS test runner States Events Requirements Add any other programmable functionModel Driven Development: Taking BDD/ATDD to the Next Level49 51. Spec Explorer Exploration Interactive model development Generate test suitesMachinesCord Declarations Model ProgramOptionsMachines define composable Action sequencesOptions control exploration and data generationActions are explored Rules define exploration conditionsModel Data StructuresRules Actions Events Adapters Events define expected SUT responseData structures support explorationAdapters abstract, control, and observe any interface, any platformModel Driven Development: Taking BDD/ATDD to the Next Level50 52. MBT Driven Development Requirements, Stories, Scenarios... ExploreAmbiguous, missing, contradictory, incorrect, redundant, incompleteTest Model RunGenerateModel error, omissionTest Suite Inputs (Test Sequences)ControlMissing, incorrect Coverage Requirements ModelEvaluateExpected Outputs (Test Oracle)SUTObserveModel Driven Development: Taking BDD/ATDD to the Next LevelBug 51 53. More Models, Less TestsMORE MODELS, LESS TESTS 54. Model-based Testing in the Real World Windows protocol compliance (Web services, RPC, DCOM) Total effort: 250 person years (mostly junior SDETs) Saved 50 person years with model-based testing Model-based Testing 1.4 Days/Requirement42% Less Time Per RequirementTraditional Testing 2.4 Days/Requirement Requirements StudyModelingTest CodingAdapter CodingModel Driven Development: Taking BDD/ATDD to the Next LevelTest Execution 53 55. Scope of Test Asset Maintenance Model-based TestingBehavior Driven DevelopmentModel How much MBTgenerated test code must be maintained?Test CodeNone!Adapters Model Driven Development: Taking BDD/ATDD to the Next Level54 56. More Models, Less Tests More Better Testing Icing the BackBlob Rule-based Maintain model Systematic and deep exploration Regenerate test suites Generate 1000s of scenarios in seconds Models grow much more slowly than test codeModel Driven Development: Taking BDD/ATDD to the Next Level55 57. Icing the Backblobhttp://www.youtube.com/watch?v=0GEJgR-bX0w 58. Q&A #MoreModelsLessTests rvbinder@sysverif.comModel Driven Development: Taking BDD/ATDD to the Next Level57 59. Learn More - MBT UCCAT (User Conference on Advanced Automated Testing) Presentationshttp://ucaat.etsi.org/2013/program_conf.html 2012 Model-based Testing User Surveyhttp://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf Open Source MBT toolshttp://www.robertvbinder.com/open-source-tools-for-model-based-testing/ Overview of MBT, list of commercial MBT toolshttp://www.slideshare.net/robertvbinder/model-basedtestingignite Spec Explorerhttp://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745 http://msdn.microsoft.com/library/ee620411.aspxMicrosoft Open Protocol Projecthttp://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#! Model Driven Development: Taking BDD/ATDD to the Next Level58 60. Learn More: BDD, etc. SpecFlowhttp://www.specflow.org/ Using SpecFlow with WatiNhttp://msdn.microsoft.com/en-us/magazine/gg490346.aspx SpecFlow Quick Starthttp://www.youtube.com/watch?v=SVwBz-ZTliw Writing Cucumber Feature Fileshttps://github.com/cucumber/cucumber/wiki Using Lava Lamps to indicate Build Statushttps://wiki.jenkins-ci.org/display/JENKINS/Hudson+Build+Status+Lava+LampsModel Driven Development: Taking BDD/ATDD to the Next Level59 61. Content and Image Sources Grand Canyon Time Lapse. Ezra Kamer http://www.youtube.com/watch?v=NF6O6HuNJ0kContrast (Order and Chaos). M. C. Escher, 1950. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270The Blob, Theatre Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=GODDLgM1gKo http://www.imdb.com/title/tt0051418/Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980Defect densities: Capers Jones, Software Quality: Analysis and Guidelines for Success Robert E. Park. Software Size Measurement: A Framework for Counting Source Statements. Technical Report CMU/SEI-92-TR-020. Software Engineering Institute, Carnegie Mellon UniversitySpec Explorer. Microsoft http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9956b-d9bfa4902745SpecFlow http://www.specflow.org/The Blob, Final Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=0GEJgR-bX0wSpec Explorer Chat Model. Microsoft.All other content Copyright Robert V. Binder, 2014.Model Driven Development: Taking BDD/ATDD to the Next Level60