agile & the pilgrimage to continuous testing

Copyright © Conformiq Inc. and its subsidiaries. All rights reserved. 1

Agile & the Pilgrimage to

Continuous Testing -

Automated Test Case Design

and Automated Test Case

Generation Robert Ashall

Softworx Technology Group

[email protected]

(o) 416-690-0730

(c) 416-464-4758

mailto:[email protected]


• Softworx - Canadian Distributor for Continuous Testing

– Optimized Test Case Generation; Test Script Generation

Automated Code Analysis & Regression Testing

Virtualize; SOAtest; JTest; C++Test; dotTEST; DTP

DevOps - Release Control & Deployment Automation

Dev/Test Environments on Demand


SDLC - The Era of Acceleration

• Constant Trade-offs that have business impact

Time

Quality

Scope


The Need for Speed is well documented

“I feel the need....the need....for speed”

Lt. Pete “Maverick” Mitchell in Top Gun


Continuous and Automated Testing is required

throughout the development process


Can we really create an automated agile process?

• What is needed?

– Tight coupling between design and component test

– Tight coupling between component and integration testing

– Automated creation of documentation for management

– Tight coupling with automated test execution

– Tight coupling with test and release management

– AND WITH SPEED!!!!

• So why can’t just any agile process work as needed for today’s complex multi-platform multi-browser products?

– Automation tools are too complex

– Tools from different vendors just aren’t tightly coupled

– People are just too slow for manual testing


Visibility Issues in Agile are Common

“If you need to develop the [agile] user story to maturity through

development, testing, and user interaction and you’re pipelining in two-

week intervals, it will probably be two months before you realize

something is wrong.”

– Joe Masters Emison, You're Not as Agile as You Think, CIO, 2013

What you can’t see WILL hurt you - so more is needed


100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

-

-

-

-

-

-

-

-

-

-

-

Using Agile and

Modeling

36.6%

35.8%

26.1%

1.5%

Not Using Agile

or Modeling

13.5%

41.1%

41.8%

3.6%

Ahead of Schedule

On schedule

Behind schedule

Don’t know

Modeling Increases Schedule Performance

Source:

VDC | Research 2016


Combine MBT Design with Test Automation

Manual

Script-Based

Capture/Replay

Frameworks

Keyword Driven

Test Models

MBT

Imp

roved T

esting

Effic

iency

Conformiq Creator™ • Automate test design

Conformiq Transformer • Simplify MBT adoption

using existing methods

• Shorten path to

test automation

• Automate test

execution


Conformiq 360O- Automate Test Design in 3 Easy Steps

The testing framework for:

Control, Visibility, Automation, Integration & Speed

Streamlined Model

Capture

1

Automatic Test Design with

One Click of a Button

2 3

Transform Generated Tests

for Automatic Execution &

Full Documentation


Step #1: Streamline Your Model Capture Activity Diagram

Activity

Decision

Data

• Model workflows in activity

diagrams

• Import diagrams from Visio, ARIS,

various BPMN tools, & Gherkin

code actions

Event

• Model available interfaces in

WYSIWYG editor

• Import interfaces from UI

recordings, WSDL/XSD, etc.

Structure Diagram

+

Modeling is not as difficult as it first appears

• "There is something confusing about the idea of good modeling being hard. After all, all one is doing is

describing how the system is supposed to work without having to worry about the implementation details. If

one can’t do that, then how is one supposed to manually create a correct, working system?“

- Thomas Mercer-Hursh, LinkedIn/Pulse, 27/10/15


Step #2: Fully Understand Your Test Cases

Generated Test

Names Individual Test

Diagram

Test Coverage

Matrix

Test Coverage

Summary

Generated Test

Case Description

Individual Test

Data

Project Control


Know Exactly What Has Been Tested

Test Generation

Information

Test Case

Coverage Test Coverage

Matrix


Most Algorithms Means Optimal Coverage

Type Description Category

Requirements Coverage Cover “requirement” statements

Basic Test

generation

Use Case Coverage Cover independently specified use cases

State Coverage Cover states of every state chart

Transition Coverage Cover transitions of every state chart

Condition Coverage Cover “true” and “false” cases of conditional constructs

Parallel Transition Coverage Cover all interleavings of independent transitions in

multi component models

Extended

Test

Generation

Data Coverage Cover all pairs or all combinations of data values

2 - Transition Coverage Cover combinations of entry/exit transitions of all states

Atomic Condition Coverage Cover all combinations that make Boolean expressions

evaluate to “true” or “false” (based on short-circuit)

Boundary Value Analysis Cover integer boundary conditions

All Paths - States Cover all arbitrarily states sequences Exhaustive

Test

Generation All Paths - Transitions Cover all arbitrarily long transition sequences

Can be used individually or in combination

On

ly C

on

form

iq

Typ

ica

l M

BT


Documentation

Custom

Foreign

Languages

Conformiq

360○ Test

Automation

ALM / Test Management

ALM / Requirements

Management

Repository

Conformiq 360o Test Automation TM

Step #3: Generate Test Scripts & Documentation

Automatic Test

Execution XML

Robot

Tricentis

TCL

Python

Perl

FitNesse

Junit

Proprietary

etc….

Existing Assets

https://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=https://ru.wikipedia.org/wiki/%D0%A4%D0%B0%D0%B9%D0%BB:Eclipse-logo.png&ei=SZxbVc_vB8OAywP7iICICg&bvm=bv.93756505,d.bGQ&psig=AFQjCNGTl7iCxxqNxdVD-pe-_nw3umqm4A&ust=1432153452528788

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.rubric.co.za/tool-support/hp&ei=WJhbVdSVOMfcywPthoDwBw&bvm=bv.93756505,d.bGQ&psig=AFQjCNGv5ArBZWDMNv0G32MUxELTncT5_A&ust=1432152523563477

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=&url=http://www.atdconference.org/&ei=SplbVYH0Luj7ywPlhYC4DA&bvm=bv.93756505,d.bGQ&psig=AFQjCNEvR3O1jcEc_Aa2_h_mkVRGwf9rZg&ust=1432152779120441

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.rubric.co.za/tool-support/hp&ei=WJhbVdSVOMfcywPthoDwBw&bvm=bv.93756505,d.bGQ&psig=AFQjCNGv5ArBZWDMNv0G32MUxELTncT5_A&ust=1432152523563477


https://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=&url=https://victorops.com/customers/&ei=pphbVYfXOuaGywOAj4DwCA&psig=AFQjCNEwFlskLcncDm2midDOAKznb2kkjA&ust=1432152615304587

https://www.bing.com/images/search?q=subversion+logo&id=EE3BC8A99F4A2E92898F39EBA4D70EFB8D1B5FEA&FORM=IQFRBA

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://developerweek.com/expo/exhibiting-companies/&ei=WpdbVdynFqe3ygPJw4D4AQ&bvm=bv.93756505,d.bGQ&psig=AFQjCNF3CS4F2hSpDFhV-R-YSK2b7ZGW0A&ust=1432152269898417

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://becuo.com/java-logo-png&ei=7JdbVY3bA4LVygPUn4DgCw&bvm=bv.93756505,d.bGQ&psig=AFQjCNGMPZHK5Jb47rCOCBgU0TfLbnCEiA&ust=1432152419558466

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.seleniumhq.org/projects/webdriver/&ei=5pVbVZ3GEsHNygPa4YOwAw&bvm=bv.93756505,d.bGQ&psig=AFQjCNHipDCBEox5jA4Ka_1XhoNmneTupg&ust=1432151904888761

https://www.bing.com/images/search?q=gherkin+language+logo&view=detailv2&qpvt=gherkin+language+logo&id=2290CA81D67229C8D6B2EC6F59C9A25D069B551A&selectedIndex=20&ccid=HUwgtm%2bC&simid=608016243031411252&thid=OIP.M1d4c20b66f825abbaff62b84d078f3aco0



https://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=https://mspartner.microsoft.com/en/us/pages/solutions/visio.aspx&ei=BBVvVY26JIWUsgGy4oL4Aw&bvm=bv.94911696,d.bGg&psig=AFQjCNFRKLurrP9oRUYmgcQGAsJDF-67sA&ust=1433429626980762

http://www.google.fi/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.keybroker.com/feed-validator/&ei=LRZvVc3RNcamsgHh942QAw&bvm=bv.94911696,d.bGg&psig=AFQjCNEdD2o9iCwnxBqxZanZKEDxBTFr_w&ust=1433429928141716

https://www.bing.com/images/search?q=gherkin+language+logo&view=detailv2&qpvt=gherkin+language+logo&id=61F6B352C7B30C4E6C84DDE609ADD5816B47814F&selectedIndex=2&ccid=Es7/CK0w&simid=608052093119563486&thid=OIP.M12ceff08ad300a6f9f0cd9f68e369b44o0


Conformiq Transformer Features

• Automate test execution without programming – Create and automate tests using drag & drop

– Import of tests generated by Conformiq Creator for execution

– Import manual tests and Gherkin scripts for automated execution

– Parallel test execution

– Data-driven test execution

– Form based test suite, test case and keyword editors

• Assisted keyword automation (user interface inspector, action auto-complete, import of recorded object repositories)

• Live-check of test suites

• Built for compatibility – Export tests to any automated test execution framework

• Consistent and uniform user interface across all execution platforms

• Execute tests against Firefox, Chrome, IE

• Analyze test execution results with HTML reports

• Utilizes user’s Selenium and QTP (& other) keyword libraries

– Integrated with Eclipse for a consolidated view for the tester


Conformiq 360 Test Automation™

Next-generation agile testing solution based on:

1. Control

2. Visibility

3. Automation

4. Integration

5. Speed

Conformiq 360 Test Automation ™

enables complex

system testing for quality at the speed of development


Conformiq in Operation Control

Model Requirements

What to test

Coverage

• Model requirements and generated tests as you go

• Control the generation of tests – Risk based, negative

and happy paths

• Automate testing in sprint

Test what you want

Test when you want

Test how you want

Delivers





http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://mac.brothersoft.com/soap-client.html&ei=nSWTVc-OMcv3oASq5Kdw&bvm=bv.96952980,d.cGU&psig=AFQjCNG8l5vKkjnJL_FPKQvQI-RVmCBZ4g&ust=1435793224787409

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAMQjRw&url=http://www.pjedesign.com/Home/About&ei=ECeTVbPzOdbqoATW0bRQ&bvm=bv.96952980,d.cGU&psig=AFQjCNGgNir31-MU3jNifPYBg8rdkSOXZQ&ust=1435793552993586



Conformiq in Operation Visibility

New Assets

Requirements

Traceability

• Visualize software actions and process

• Easily see what every test case covers

• Comprehend end-to-end flow

Visual Coverage

Pos/Neg Paths

Map Tests to Rqmts

Delivers

New & Existing Assets Test Case

Coverage






http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://mac.brothersoft.com/soap-client.html&ei=nSWTVc-OMcv3oASq5Kdw&bvm=bv.96952980,d.cGU&psig=AFQjCNG8l5vKkjnJL_FPKQvQI-RVmCBZ4g&ust=1435793224787409

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAMQjRw&url=http://www.pjedesign.com/Home/About&ei=ECeTVbPzOdbqoATW0bRQ&bvm=bv.96952980,d.cGU&psig=AFQjCNGgNir31-MU3jNifPYBg8rdkSOXZQ&ust=1435793552993586

https://www.bing.com/images/search?q=gherkin+language+logo&view=detailv2&qpvt=gherkin+language+logo&id=61F6B352C7B30C4E6C84DDE609ADD5816B47814F&selectedIndex=2&ccid=Es7/CK0w&simid=608052093119563486&thid=OIP.M12ceff08ad300a6f9f0cd9f68e369b44o0


Conformiq in Operation Automation

• Determine invalid tests automatically

• Automatically generate from models scripts & oracle for

execution

• Automate existing manual test cases

• Always current documentation

Automatic test

script generation

Correct results

Optimize regression

pack

Delivers

Automatic Test

Case Creation

from Models

Automatic Test

Execution

Automatic

Documentation



Requirements Management

Modeling Assets

ALM /

Test Management

• Totally integrated for fully automated testing

• Import & automatically verify requirements coverage

• Import interfaces from UI recordings, WSDL/XSD, etc.

• Import models BPM or custom (iRise, Magic Draw, …)

• Export test cases to test management

Leverage your

SDLC framework

Synchronized

development & test

Delivers Export

Test Cases

Conformiq in Operation Integration

Links to Automated

Execution



Conformiq in Operation Speed

• Automatically yield minimum tests & maximum coverage

• Automatically utilize multiple processors & cloud-based

resources to create tests

• Reuse models as “IP” building blocks

• Use existing manual test cases for automation & models

Fewer tests

Greater coverage

Parallel computing

Easy model reuse

Automated manual

Delivers

Most Advanced

Algorithms &

Generation Engine

Multi-Core Processing

Reuse & Combine Models



Complete Automation of the End to End Testing Process

Conformiq 360O

– A Total Testing Transformation

Modeling

Business

Process

Management System

Under

Test

Automated

Test

Execution

Test

Management

Requirement

Management

Version

Control Manual

Tests

Test Design

Screen

Recording

Creator Transformer


Questions?

Robert Ashall

Softworx Technology Group

[email protected]

(o) 416-690-0730

(c) 416-464-4758

mailto:[email protected]

agile & the pilgrimage to continuous testing

Documents