devops - shift left - ibm · integration testing throughout your project lifecycle –with a...

© 2013 IBM Corporation

DevOps - Shift Left

Shift Left to better business results

Continuous TestingDelivering higher quality software faster by shifting left

Glyn Rhodes – Product Manager

1 © 2013 IBM Corporation

IBM Rational Test Workbench

– Enable functional, performance and

integration testing throughout your project

lifecycle

– With a scriptless, wizard-driven test

authoring environment and support for

more than 70 technologies and protocols

IBM Rational Performance Test Server

– Save time and effort by reusing your

functional integration tests for performance

testing

– Combine your tests to model real-world

scenarios and assess the impact of load on

your integrated infrastructure.

IBM Rational Test Virtualization Server

– Model real system behavior to eliminate

test dependencies and simulate production,

– Accelerate testing and reduce the setup

and infrastructure costs of traditional testing

environments.

Test

Workbench

Functional Testing

Integration Testing

Performance Testing

Regression Testing

Mobile Testing

Test

Servers

Load Agents

Virtualization Agents

QA

Test Lab

Infrastructure

DEV

The Continuous Testing Solution: RTW, RPTS, RTVS


Without continuous testing there

is no continuous delivery!

Continuous Testing:

•Helps extend agile and lean practices across the

delivery lifecycle

•Removes traditional testing bottlenecks such as

unavailable test environments to increase

efficiency

•Enables teams to test earlier and with greater

coverage at lower cost

•Provides project teams with continuous feedback

on software quality to reduce business risk

•Avoid the big bang!

Why organizations need Continuous TestingAccelerating the delivery of higher quality softwareBetter – Cheaper - Faster

Resolve today’s testing

challenges with confidence.


Development/Test - Expectations continue to increaseBetter – Cheaper - Faster

In the Past

•Organizations looked to find a

balance between speed, cost & risk.

•Development teams were not as

agile as they are today; testing

never seemed to have enough time.

Today

•Organizations and customers are

demanding teams release higher

quality software faster.

•Development teams are becoming

more and more agile; testing still

never seems to have enough time.


The reality is…Current approaches are not helping

Testing Labor

Outsourcing labor is no longer

a sustainable model as global

wages are increasing

Other costs

Testing costs

Development VelocityTest Lab Complexity

Product and application

complexity are driving

test lab cost increases

Test teams can no longer

keep up with agile

development

Heterogeneous Environments

Public CloudPrivate Cloud

Data Warehouse MainframeEnterprise

Service Bus

Directory Identity

File systems

Collaboration

Web/Internet

Routing Service

Third-partyServices Portals

ContentProviders

EJB

SharedServicesArchives

Business Partners

Messaging Services

Traditional

Testing


IBM DevOps – Continuous TestingAdopting continuous testing principles provides early and constant feedback on quality

Accelerate

software delivery –

for faster time to value

Balance speed, cost,

quality and risk –

for increased capacity

to innovate

Reduce time to

customer feedback –

for improved customer

experience

Continuous

Customer

Feedback &

Optimization

Collaborative

Development

Continuous Release

and Deployment

Continuous

Monitoring

Continuous

Business Planning

Continuous

Testing

Operate Develop/ Test

Deploy

Steer

DevOpsContinuous Feedback

Our DevOps POV is resonating with clients and they are delivering

measurable business outcomes with DevOps


Dev/Test and Ops – mutual goals


Dev and Test – mutual goals


Business drivers for change

• Market opportunity

• Keeping ahead of the competition

• Customer satisfaction

• Technology platform for future growth opportunities


The rise of complex, distributed systems

• Built for interoperability

• Various technologies

• Distributed (inc. 3 party)

• Contention

• We need to test them!

30-50% average timeTesting teams spend on setting up test

environments, instead of testing

$5-30 million investmentTypically required to build a single test

lab for a Fortune 500 company. Most

have dozens…


What does Shift Left mean for testing?

Development Released

Go Live

If you are testing then you have

already begun the journey

When defects are

found


The baseline

Development

When defects are

found


Today, integration of enterprise applications still follow the “big-bang” approach …

Integration amongst different components of the enterprise application occur at the end of a milestone in a “big-bang”

Difficult to continuously integrate and deploy across different technology stacks and varying delivery schedules. Very hard for different teams to collaborate together on interim builds inside a milestone

Mobile

component

Web

component

Mainframe

component

Inte

gra

te

typ. 6 Weeks

1 week

Big Bang!

Inte

gra

te

typ. 6 Weeks

1 week

Big Bang!


The Gap

Integration TestingUnit Testing E2E Testing


New world of continuous testing with shift left

Integration Testing

Unit Testing

E2E Testing

Performance Testing

UAT

• Automated integration testing fill the gap…

• … and shifts to the left to execute just after unit testing

begins

• More on performance testing later


The changing role of the tester

Technical Tester/Dev Traditional Tester

Performance Testing UI Testing

Executing Testing Frameworks

Acceptance Testing

Unit Testing

Building Testing Frameworks

DevOps Tester

Integration Testing

Service Virtualization

Continuous Testing


Test Centers of Excellence need to change too

TCoE

Project A

Project B

Project C

Project D

• Promote culture of flexibility

• Eliminates barriers to progress

• Central asset repository

• Skills the workforce in automation

and service virtualization


Shift Left with Automated Integration Testing

Tests can bypass the UI and execute

beneath the iceberg. Test while your

system is being built and at an

accelerated rate. Much greater coverage

= much greater confidence


Shift Left with Service VirtualizationA shift in your test environment strategy

• IBM technology helps to eliminate project and environment contention. Create virtual components that faithfully simulate behavior of missing components

• Now you can test when environments are unavailable or even before they have been built

• A fundamental change to the way you can approach your testing

Your system

Interfaces to

other

systems

(e.g. Billing)

Substitute with

virtual

component

Continue to test even

when all interfacing

systems are not available

This new testing approach is called “Service Virtualization”


An Example

Finance

Sales

Manufacturing

CRM

Transport

HR

HTTP

SAP RFC

Tibco

MQ JMS

Batch File

JDBC

MQMQ

SAP IDOC

SAP RFC

SAP XI 3.0

SAP BAPI

SAP IDOC

HTTP

HTTP

HTTPSJDBC

SAP BAPI

HTTP


Presentation layer

Integrations, data and business logic

Only one-tenth of an iceberg is above water

Traditional UI testing:

Automated Integration Testing and Service Virtualization:

The majority of risk in modern systems is seldom tested as it is unseen.


Discover

Client

Config

Server

Web

Service

What kind of services do you have?

Here’s the list and how to work with

them.

HTTP(s)

XML REST

SOAP JSON

XSD

WSDL

WADL

Accelerator

Accelerator

• Record transactions

• Or use accelerators such as WSDLs to short cut the process

• Build tests and stubs across a range of supported protocols and formats


Example details

Mantras:

• Test what changes, virtualize what does not change

• Prioritize the most technical tests

• Plan to be flexible

Steps:

1. Discover

2. Build Tests and Virtualized Components

3. Execute

4. Incrementally increase scope

Sales

CRM

Transport

HR

MQ

HTTP

HTTP

Project A

Project B

Project CNo Change


The Critical Path for Continuous Testing

Project A – 100 days

Project B - 20 days

Project C – 30 days

1

2

• The critical path is 100 days – any delays from Projects B and C will delay the program

• The impact of late delivery from Project C is much greater as the burn rate from the

program is greater at that point

• Failure at a technical integration level between the projects is the greatest risk

Burn rate

$20k per

day

Burn rate

$75k per

day

1. Project B is at risk of running late. Test environments for integration between A and

B are not ready. Every day it runs late, $20k will be burned, and the critical path is

pushed out.

2. Project C is running late. They are only half-way through their testing and are not

able to integrate yet. Every day it runs late, $75k will be burned.


Service Virtualization across the SDLC

Define constrained

systems and

services for

virtualization

Integration OperabilityUser AcceptanceSystem IntegrationSystemUnit Integration OperabilityUser AcceptanceSystem IntegrationSystemUnit

Performance

Developer uses

virtual services to

unit test code on

their workstation –

no elaborate

environment

required

Development and Test teams

uses virtual services for early

stage advanced testing

The full system or components

of the system can be

performance tested early in the

cycle. Virtual services can be

used when components are

not available

Later in the cycle, final pre-

production versions of services

are swapped in as they

become available. Final testing

is done against real services


DevOps Automated Testing MaturityFor complex, distributed environments

- UI testing

- Some Integration Testing (mainly manual)

- Early and continuous integration testing

- No “big bang” (1)

- Reactive service virtualization

- Traditional performance testing

- Early and continuous performance testing

- Plan ahead with service virtualization (2)

- Automated environment provisioning

- E2E orchestration between UI and integration testing

- Automated data provisioning

Beginner Intermediate Advanced Extreme

1. Don’t wait until all components are ready before beginning

integration testing

2. Test Managers can plan ahead and mitigate the risk of

environments not being available


Continuous Deployment and Testing

Change

Automated Deployment

Automated Testing

FeedbackFeedback

into dev, test

and ops

Deploy real

and

virtualized

components

Integration

testing and

UI testing


Continuous testing with virtualized services

Automating deployment and testing to production-like environments

Databases Mainframeapplications

Third-partyServices

Rational Test Virtualization Server

Application

changes

being tested

virtualized services

Dev QA

Rational

Team Concert

Jenkins

Change

Rational Test Workbench

Test

IBM UrbanCode Deploy

Feedback


Continuous Performance Testing

MONITORING

PREDICTIVE

ASSESSMENT

ASSESSMENT

Low Intensity

Performance Testing

(LIP)

High Intensity

Performance Testing

(HIP)

Incremental integration strategy defines HIP start point;

focus of HIP is informed by LIP results

Requirements Design Coding SIT UAT Production


Requirements

Big Bang

E2E testing UATIntegration testing

£290

£1032

£97

Reduction

in cost of

fixing

defects

£16 Unit test

£97 Integration test

£290 E2E Test

£1032 in UAT

Source: SEI

Measuring success with defect removal efficiency


Summary

Getting to market faster requires a collaborative and continuous approach to testing

Continuous testing aims to fill the gap with automated integration testing…

…and shift it to the left

Discovery becomes a core skill for the DevOps tester

Complex and high risk tests can be prioritized

Plan to be flexible with service virtualization

Continuous testing coupled with continuous deployment, checks the software & validates the deployment process for ops


Shift LeftIs your effort directed at your risk?

UNIT SERVICE UI

Automated Integration

Testing

Service Virtualization

Traditional UI Tools

Testing only at the UI level provides little confidence for deployment of complex distributed systems

Testing must be shifted left to focus on the highest areas of risk

Automated integration massively increases testing coverage and confidence

Service virtualization can be applied at all phases – even allowing integration testing to begin in parallel with Unit Testing.

Shift Left


Additional ResourcesContinue your learning journey as you shift left

ibm.co/ARDfordummies

ibm.co/ServiceVirtualizationForDummies

ibm.co/agilefordummies

ibm.co/devopsfordummies


Additional ResourcesContinue your learning journey as you shift left

http://www-01.ibm.com/software/rational/servicevirtualization/

YouTube playlist:

https://www.youtube.com/playlist?list=PLlhg84-xzjfwdUShtwq7sG4HZQpFUwwFM

Search for: IBM Service Virtualization

http://www-01.ibm.com/software/rational/servicevirtualization/

https://www.youtube.com/playlist?list=PLlhg84-xzjfwdUShtwq7sG4HZQpFUwwFM

34

ibm.com/devops

http://www.ibm.com/devops


Acknowledgements and disclaimers

© Copyright IBM Corporation 2012. All rights reserved.

– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products and

services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these

and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate

U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or

common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at

www.ibm.com/legal/copytrade.shtml

Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries

in which IBM operates.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for

informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant.

While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without

warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this

presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or

representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of

IBM software.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have

achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to,

nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

http://www.ibm.com/legal/copytrade.shtml

devops - shift left - ibm · integration testing throughout your project lifecycle –with a...

Documents