test automation: investment today pays back tomorrow

!

!

T11#Session!6/25/2015! !1:30!PM!!!!!

“Test#Automation:#Investment#Today#Pays#Back#Tomorrow”##

Presented#by:#

Al#Wagner#

IBM######

Brought#to#you#by:#####

##

340!Corporate!Way,!Suite!300,!Orange!Park,[email protected]!D!www.sqe.com!

!

!!!!

!!!!! !!!!!

Al Wagner is currently a testing evangelist with IBM, driving thought leadership, strategic initiatives, and tangible solutions with a specific focus on quality management, test automation, and service virtualization. In his fifteen years of practical IT field experience, Al has assisted, supported, mentored, and enabled teams with their IT application infrastructure, development, implementation, and operations challenges. A frequent conference speaker on software quality principles and techniques, Al has coauthored the IBM Service Virtualization for Dummies book and authored numerous papers on quality management and test automation.

!

!

Al Wagner IBM

5/26/2015

1

© 2013 IBM Corporation

Continuous delivery of software innovation on hybrid cloud

Test Automation:Investment today pays back tomorrow

@Al_WagnerIBM Testing [email protected]

1

Today’s reality for Testers

Testers want to spend MORE time

ibm.biz/TestingSurveyResults

5/26/2015

2

2

Are we simply chasing Rainbows and Unicorns?

3

Test Automationis not a “Silver Bullet”!

5/26/2015

3

4

Not all tests can or should be automated� Where test automation makes sense

� Highly repetitive tasks� Well understood test cases or scenarios� Stable areas of the application under test

� Where test automation doesn’t make sense� Tests which need that human component

• Exploratory tests� “One off” tests� Experimental tests

5

A dream doesn’t become reality through magic; it takes sweat, determination and hard work.

-Colin Powell

5/26/2015

4

6

There is no “Easy” button� Test automation is a Software Development project

� Needs funding� Needs resources� Needs planning

� Automated tests need to be designed, developed and tested� Build a reusable code base – framework� Take an iterative approach – learn and improve� Modularize your automated tests - reuse

� Test automation is a Development Project that does not end� Maintenance� Ongoing development & refactoring� Continuous improvement

7

Test automation tools are only robots

� Test Automation tools are not the same as human Testers…� They do not think, ask questions, challenge results� They do not get tired� They are not impacted by other interruptions

� Test Automation tools excel at…� Executing a series of steps the same way repeatedly� Reporting on programed expected results� Speeding up test execution (the “checking” of software)

Test automation should not be an “excuse” to reduce head count!

5/26/2015

5

8

Not all Testers are Automation Engineersunderstand how to best leverage the existing skills of your team

� Subject Matter Experts• Understand the needs of the business – Domain Knowledge• Aide in making sure software being developed is testable• Assist in designing and reviewing test cases

� Automation Engineers• Understand OO Concepts

• Encapsulation, Abstraction, Inheritance, Polymorphism• Have a development background

• knowledge of programming/scripting languagesJava, Jscript, PHP, Ruby, Python, Jquery, etc…

• Build and maintain test automation assets

9

With the right approach & skilled resources

Test automation can…� Help decrease the duration of a test cycle

� others have decreased their test cycle from months to hours

� Increase frequency of test execution� being able to run faster means you can run continuously

� Improve level of testing by freeing up time� planning and design; additional value-add tests

� Provide earlier detection and isolation of defects � decrease project risk

� Improve confidence with software quality� increase test coverage; improve the quality of testing itself

5/26/2015

6

10

Automating tests – Historically speaking

How users typically approachautomation…- Record and playback

Automated UI tests are… • the most fragile • the most difficult &

time consuming to maintain

as the application continuously changes.

“Record & Playback”of the User Interface

Integration Tests

Testing messages exchangedbetween components

Unit TestsTesting the lowest

level of an application –a component

and itsmethods

11

Automating tests – Is there a better way?

What the user can see…- User Interfaces- Systems of Engagement

What’s below the surface…- Integrations- Components- Objects/Classes- Methods- Systems of Record- Infrastructure- and much more

UI Testing

5/26/2015

7

12

Take a better approach to automating tests

Exploratory Testing

ComponentLayer

Service / APILayer

UILayer

Push testing down to focus on the highest

areas of risk

Automated integration testing increases

coverage and confidence

Service virtualization can be applied across

all layers

13

Test what matters, test it earlier, test it continuously

� Earlier team collaboration focuses on prevention over detection

� Test execution shifts left with focus on testing the highest areas of risk

� Automated integration testing increases testing coverage and confidence

� Service virtualization can be applied at all phases – even allowing Integration Testing to begin in parallel with Unit Testing.

UNIT SERVICE UI

Automated Integration Testing

Service Virtualization

TraditionalUI Testing

Development Testing Staging Production

Collaboration

5/26/2015

8

14

Service / API testing (aka Integration Testing)

� Tests can bypass the UI and execute at the point of integration

� Test while your system is being built and at an accelerated rate

� Much greater coverage = much greater confidence

15

Comprehensive test automation

Continuously Test in Multi-Channel, Complex and Distributed Environments� Functional, Integration, UI Testing, and Mobile Testing

� Test across multi channel integrated systems

� Test across platforms and technologies

� Execute full scale, orchestrated performance testing, or test performance at the integration layer

Host/Mainframe

Client-Server

Web/Desktop

Mobile/Cloud

5/26/2015

9

16

Testing the UISeparate what the test accomplishes from how it is executed

�Construct test scripts which are…durable, maintainable, resilient to change�Keywords abstract the details of how to execute a logical unit of function�Business scenarios compose keywords into a flow

� Reduce test maintenance�Keyword behavior isn’t reproduced in multiple

tests; update fewer places when application changes

�Reduce test cases because overlap becomes apparent when abstraction is raised to keyword level

17

Test Script1)2)3)4) Review shopping

cart contents5) Remove item

from cart6) Verify removal7)8) Logout

Logon & Verify

Click Order Entry

Enter Order Details

Click Home




Logon & Verify

Click Order Entry

Enter Order Details

Click Home

Automating Tests – Maximize reuse / Minimize maintenance

Test Script

Test ScriptTest Script1) Logon & Verify2) Click order entry3) Enter order

details4) Click process

shopping cart5) Enter billing info6) Submit order7) Verify order

confirmation8) Click Home9) Logout

Test Script1) Logon & Verify2) Click order entry3) Enter order

details4) Click review

shopping cart5) Remove item

from shopping cart

6) Verify removal7) Click Home8) Logout

Test Script1)2)3)4)5) Enter billing info6)7) Verify Order8)9)

Keyword Repository

Logon & VerifyClick Order Entry

Enter Order Details Process Cart

Click Home

Logout

Logon & Verify

Click Order Entry

Enter Order Details

Submit Order

Click Home

Process Cart




Logon & Verify

Click Order Entry

Enter Order Details

Click Home

Logout

Submit Order

Modular ApproachTypical Approach

Easier to create; Offers minimal reuse; Requires maximum maintenance

Harder to create; Offers maximum reuse; Requires minimum maintenance

5/26/2015

10

18

Automated Tests!=

Automated Testing

19

Automated testing enables continuous testing

AutomatedTests

AutomatedTesting

5/26/2015

11

20

Quality Management across the Lifecycle

�Build quality in with team collaboration� Unify the team through real-time collaboration� Minimize scrap and rework with lifecycle traceability

�Manage risk with smarter testing� Focus your testing effort on the right 20%� Apply automation where it matters

�Make confident decisions� Take informed decisions and make

proactive change with real-time analysis and actionable reporting

� Confidently deliver incremental quality improvements

21

Today’s reality for Testers

Testers want to spend LESS time

ibm.biz/TestingSurveyResults

5/26/2015

12

22

Service virtualization

� Eliminate traditional project and environment contention � Create virtual components that realistically simulate the

behavior of missing components� Test even when environments are unavailable,

resources are unavailable – or before they have been built

� Change the fundamental way you approach testing

Your system

Interfaces to other

systems (e.g. Billing)

Substitute with virtual

component

23

Public CloudA public cloud is one based on the standard cloud computing model, in which a service provider makes resources, such as applications and storage, available to the general public over the Internet.

Public cloud services may be free or offered on a pay-per-usage model.

Private CloudPrivate cloud is the phrase used to describe a cloudcomputing platform that is implemented within the corporate firewall, under the control of the IT department.

Hybrid CloudA hybrid cloud is an integrated cloudservice utilizing both private and public clouds to perform distinct functions within the same organization.

• Cloud is not a commodity. • Cloud computing doesn’t come out of the sky. • Clouds are physical hardware inside brick and

mortar facilities connected by hundreds of miles of networking cable.

Dynamic infrastructurecomputing and networking resources available on demand

5/26/2015

13

24

Continuous deploymentautomating testing includes automating deployment� Consistent deployments and rollbacks of applications� Orchestration of changes across servers, tiers and components� Configuration and security differences across environments� Clear visibility: what is deployed where and who changed what

• Multi-Tier Application Models• Scalable distributed automation• Quality gates and approvals

Deployment Automation

25

Our “sample” application!

Mobile Web UI

Database

SAP

CICS

ApplicationServer

Android, iOS IE, Firefox, Chrome, Safari

MyFictiousCompany.com

Dependent3rd party service

MyFictiousCompany App

5/26/2015

14

26

The “many” environments of MyFictiousCompany

Deployment Automation

MobileEnv2

AppServer

3rd PartyService

DatabaseCICSSAP

Physical devices, emulators, devices clouds

Rational TestVirtualization

Server

- virtualized services

CICSTx

SAP ExternalServices

DB

ServiceVirtualization

Development ProductionWeb UI Env1

DB

AppServer

SAP

CICS

PaaS

3rd PartyService

TestCustomers

DB

AppServer

SAP

CICS

PaaS(private cloud)

External Services

SaaS(public cloud)

Off premise

Off premiseOn premise

27

CollaborativeDevelopment

Continuous Feedback

In a DevOps worldTesting and test automation is critical to accelerating software delivery

The Continuous Delivery Pipeline

Development Testing Staging ProductionBusiness Owners Customers

ContinuousMonitoring

Continuous Testing

Dev CI

At least 1/3 of is testing!

Quality Managementacross the lifecycle

Production like Test Labs

Comprehensive Test Automation

Orchestration frombuild to production

5/26/2015

15

28

Continuous Customer Feedback & Optimization

Collaborative Development

Continuous Release and Deployment

ContinuousMonitoring

ContinuousBusiness Planning

ContinuousTesting

Operate Develop/ Test

Deploy

Steer

DevOps Continuous Feedback

Acceleratesoftware delivery –

for faster time to value

Balance speed, cost, quality and risk –

for increased capacity to innovate

Reduce time to customer feedback –for improved customer

experience

The Big Picture - DevOps & Continuous Testing

Clients are achieving measurable business outcomes with DevOps!

© 2014 IBM Corporation

Enterprise capability for continuous software delivery that enables organizations to seize market opportunities and reduce time to customer feedback.

29

Paybackis Real!Real Results from Real Projects

5/26/2015

16

30

Results of adopting Test Automation & Service Virtualization

A Major European Bank

� Reduced test cycle from 3 weeks down to 4 hours� Increased project delivery capacity 100% (40 projects to 80 projects)� Avoided hiring three additional FTE's, saving almost $600k.� Reduced incidents found in production from 2.5 to .3 per project, saving $3 million� Identified defects earlier and freed the team up to focus on new capabilities, saving over

$560k� Reduced consulting and third party testing fees with better test automation, saving

$408k� Eliminated the need for additional hardware and labor to set up test environments,

saving $700k� Generated new revenue for the business – estimated at tens of millions!

13x ROI in 3 years!

Investment$433,149

Present Value$5,839,998

31

“Test automation reduces the time required to validate ‘rate filing’ by 94 percent, from 5,600 hours to 320 hours, and prior to this software, manual testing represented 67 percent of the total rate filing time.”

An insurance provider

Results from automating testinggoing where others have already gone

5/26/2015

17

32

“The new framework helped to drive new efficiencies, cutting deployment timelines for new tools and reducing defective code within mobile-friendly systems by up to 90 percent.”

IBM CIO Office


33

“Sandhata Technologies Ltd. cut costs by $100,000 per project by simplifying functional and regression testing for more rapid project delivery. Test automation accelerated development into production, lowered the risk of undertaking change and reduced the costs of running the project.”

Sandhata Technologies Ltd.


5/26/2015

18

34

“Companies testing their software through SQUALIO have been able to reduce their software testing costs by 30 percent due to improvements in infrastructure expenses and workflow efficiency.”

SQUALIO


35

Summary� Do something…

� It is better than doing nothing. What have you got to lose?� Remember that automated tests are software…

� Automated test scripts and frameworks need on-going care� Adopt a framework approach…

� Develop scripts which are easy to maintain and resilient to change� Don’t be afraid to experiment…

� We learn from our mistakes � Focus on pushing automating tests down….

� To the component and service layers� Automating tests != automating testing…

� They are not the same� You are not the first…

� many have experienced success; be that next one!

5/26/2015

19

36

Training accelerates Productivity

Don’t let yourself be just another “fool” with a “tool”!

�Formal Training� face to face� virtual

�Mentoring�Pairing� and many more

37

Help is available!!!

� test

5/26/2015

20

38

IBM Positioned As A LeaderThe Forrester Wave™: Modern Application Functional Test Automation Tools, Q2 2015Ensure Quality Of Enterprise, Mobile, And Web Applications Without Sacrificing Delivery Speed

“In order to support continuous quality and faster development, especially for those on an Agile and DevOps journey, testing must shift from a UI-only approach to an UI-and-API approach…”

The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical representation of Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change.

External Presentation Slide

ibm.biz/ForresterWave2015Test

39

5/26/2015

21

40