grid-tools / ca - reactive automation - moving from requirements to automation

Reactive Automation Moving from Requirements to Automation

Huw Price, CA TechnologiesJonathon Wright, Hitachi Consulting

03/03/2016

2 © 2016 CA. ALL RIGHTS RESERVED.

FeaturingHuw Price

VP, CA Technologies, Inc.

Jonathon Wright

Director, Hitachi Consulting


Most testing is – Frankly:

Random

Unstructured

Repetitive

Not thorough enough

Can’t be measured

Can’t keep up

Too slow

Model based testing lets you define what is supposed to happen and then test that.

Model based testing is:

Accurate

Structured

Thorough

Measurable

Can keep up with change

Allows you to measure risk and resources

Lets you automate very quickly

Model Based Testing


A CA pre-webinar survey, conducted July 2015 (112 responses)

0% 10% 20% 30% 40% 50% 60% 70%

Time/resources in test data compliance (PII)

Defects stemming from ambiguous requirements

Testing innefficiencies leading to higher cost

Lack of test coverage creating defects/rework

Difficulty finding the right data for a particular test

Manual testing leading to project delays

What are the main software challenges you are facing? Select all that apply.


Pre-webinar survey – question 2 (97 responses)

0% 10% 20% 30% 40% 50% 60% 70%

Test Data Warehouse and Test Data Allocation

Data Masking and subsetting

Synthetic Data Generation

Requirements Definition

Test Case Optimization

Automated Test Case Design

Test Automation

In what ways do you consider (or have already started) using Test Data Management and/or Test Case Design

technology for? Select all that apply.


Pre-webinar survey – question 3 (99 responses)

0% 10% 20% 30% 40% 50% 60% 70% 80%

Meet data compliance requirements

Automated creation and execution

Optimized Test Data Coverage

Reduced testing cost and complexity

Accelerated Time to Market

Improved software quality

Expected/realized benefits of TDM and TCD


Addressing the ‘Digital Enterprise’ imperative

* Hitachi Consulting, Becoming a Digital Enterprise: www.hitachiconsulting.com/digitalenterprise


Requirements are static, incomplete and ambiguous

Over 50% of defects are introduced in the design phase


Poor requirements

A plethora of requirements techniques exist

They tend to create ambiguous, incomplete

requirements, creating defects

56% of defects stem from ambiguity in

requirements1

1 – Bender RBT, Requirements Based Testing Process Overview, 2009

http://benderrbt.com/Bender-Requirements Based Testing Process Overview.pdf


Requirements and change requests are usually a “wall of words” or static diagrams

The requirements are “static” - they

offer no way to derive tests directly

from them…

…And no way to update tests when

the requirements change – this has

to be done manually


The Problem:A lack of clarity and vision during development

Business Analyst Programmer

TesterUser

The User Knows what they want

The Analyst specifies what that is

The Programmer writes the code

The Tester tests the program


The Solution:Clarity and Vision during development

Business Analyst Programmer

TesterUser

There are less bugs and the product is delivered faster

The closer the vision means the user gets a quality product


* Hitachi Consulting, Engineering the New Reality, www.hitachiconsulting.com/newreality


Digital Evolution = #DesignOps

Build

Deliver

MonitorMeasure

Learn

Design

Make

Check

Think

Digital Business Model

AdaptiveIT

LeanUX

Design

PivotEvolve

DevOps

DisruptInnovate

LeanIT

Operations


Predictive Improvement

Predictive Learning

Predictive Intelligence

Predictive Insight

Predictive Assessment

Predictive Quality

Predictive Innovation

Predictive Testing

Predictive Delivery

Predictive Support

Predictive Experience

Predictive Operations

DIGITAL AT THE HEARTDIGITAL PROCESSES LEANDIGITAL TECHNOLOGY DESIGNOPS

Technology Processes Behaviours


Test Automation is not a Silver Bullet

Automated testing frameworks are heavily scripted

Script generation is usually done manually

As well as the maintenance

Alternative solutions, use:

Record Playback

Use script-less automation frameworks (keyword)

But you’re back to Manual test case design

Automated tests: manual generation


Manual Test Case Design is slow and unsystematic

Currently manual – a time consuming, error-prone process

Is unsystematic, ad hoc, and has no real notion of “coverage”

Over-testing and under-testing – 10-20% coverage with 4 times over-testing

Poor requirements lead to poor overall testing, with testers having to fill in the gaps

No linkage to test data – process is manual, painstaking and very time-consuming

No flexibility for change requests: a critical weakness in an agile or Continuous Delivery environment. Changes take longer than the original requirement!


Data is not linked to tests and testers have to sieve through high-volume, low-variety production data sets, which cover just 10-20% of tests

20% of the SDLC is spent waiting for data

Data constraints force testers to wait for data to become available ‘upstream’

Data is not available in parallel, across teams, projects or releases

A change made to data effects every team, so that tests fail for apparently no reason, and data refreshes take days or weeks

The right data is never available when testers need it


To Create Perfect Test

Cases

To Manage Test Data

To Manage change in Test Cases

To Create Automation

Scripts

From One Input

Create Multiple Outputs

=

Less Language Hops

Less ProductHops

To Estimate Complexity

Populate Story

boards & backlogs

What value does Agile Designer provide?

To Build betterRequirements

To Improve my Existing Test Cases

To Manage SV

FasterBetter

Cheaper


• Test cases and scripts are created automatically from “Active” requirements

• They are executed automatically

• Testing is Model Based

• Use a Component Library of common and optimized tests

• Test Data and Virtual End Points are created or found as part of the Automation

• Automating the automation

Active Automation

Active Automation


Model requirements as an “Active” Flowchart

A formal model that is accessible to the business

who already use VISIO, BPM, etc.

Which is also a mathematically precise model of a system, so that it eliminates ambiguity and incompleteness

It can be used by testers and developers – it brings

the user, business and IT into close alignment


The “Active” Flowchart

Testers can overlay the flowchart with all the functional logic and data involved in a system

Tests can therefore be automatically derived from it


Auto-generate test cases directly from requirements

Test cases can be created automatically, in minutes – not days or weeks

They are optimized, so that they test 100% of functional coverage in the smallest amount of tests possible


Auto-generate test cases directly from requirements

Complexity and coverage can be measured

Tests can be executed as either manual tests, or automated tests


“Match” Tests Directly to the Right Data and Expected Results “Match Jobs” can be run automatically, finding data from multiple back-end

systems, the Test Data Warehouse, or creating it from scratch when none exists

Data is created from default values, based on the output names, attributes and values defined in the flowchart

Expected results are linked to the logic gate in the flowchart, and are exported along with the test cases


Generate all the test data you need

1. Automatically profile data, model it, and accurately measure its coverage

2. Generate rich synthetic data which provides 100% coverage

3. Cover every outlier, unexpected result, boundary condition and negative path

4. Create thousands of rows of complex, inter-related data in minutes

“Empty”

Datamaker + Required Data Characteristics

Provision fit for purpose data anytime and every time!Provision data with or without access to production systems!

Ready for Testing!


Full traceability with requirements - Auto-update tests when the user requirements change Know what needs to be re-tested and when the integrity of a system is at risk… “If I

change this, what will I break?”

The impact of a change made to an individual component is identified system wide

The impact on test cases and userstories up and down a system

can also be identified automatically


Update manual or automated tests in minutes

Remove any broken or redundant tests automatically – no more checking test cases by hand

Restore functional coverage to 100%, creating any new test cases required

Execute only the tests needed to validate a change

Make sure a change has successfully “rippled up”


Active Automation

Record an Automation Script

Import it into Agile Designer

Design the logic

Auto create your automation robotsIf something changes, auto create new robots!

Cl Cl

Cl

ClN

O

H

N

N

N

H

O

Cl

Cl

Cl Cl

Automate Change


Model Based Testing – Use Libraries of Test Cases


Case Studies - Power Station


Power Station – FMEA & Fault Tree Analysis

Examine the data for the Fukushima Nuclear

incident and create the fault tree that relates to the

accident. Determine what could have been done to

prevent the accident and avoid the Undesired

Event which is the prevention of Level 7 Nuclear

Incident

Fault tree analysis is a technique that used

Boolean logic to describe the combinations of

intermediate causal effects that can initiate a

failure. Unlike FMEA FTA starts with a specific

failure and strives to enumerate all the causes of

that event and their relationships. A fully

constructed fault tree represents a failure and all of

it’s potential causes.> P(A or B) = P(A B) =P (A) + P(B) – P(A B)


Virtual Power Plant – Enterprise of Things


Virtual Power Plant – Digital Broker

38

A virtual power plant is a link-up of small, distributed power stations, like

wind farms, photovoltaic systems, small hydropower plants and biogas units

that can be switched off, in order to form an integrated network.


Virtual Power Plant – Predictive Weather

WS2

WS1

WS3

WS4

39

AB

C

39393939

“There have been some estimates from Cisco that there will be more than 50 billion objects connected to the Internet by the year 2020. Jonathon Wright has often saidthat when we talk about IoT we are talking about hardware -- not software.”

D


CA Test Case Optimizer, Test Data Manager, Agile Central and HPE ALM


[email protected]

slideshare.net/CAinc

linkedin.com/company/ca-technologies

VP, Test Data Management

Huw Price

@datainventor

Director of Digital Engineering, Hitachi Consulting

[email protected]

Jonathon Wright

@Jonathon_Wright

linkedin.com/in/automation

hitachiconsulting.com

Contact Us

slideshare.net/Jonathon_Wright

ca.com