continuous testing through service virtualization
DESCRIPTION
The demand to accelerate software delivery and for teams to continuously test and release high quality software sooner has never been greater. However, whether your release strategy is based on schedule or quality, the entire delivery process hits the wall when agility stops at testing. When software or services that are part of the delivered system, or required environments are unavailable for testing, the entire team suffers. Al Wagner explains how to remove these testing interruptions, decrease project risk, and release higher quality software sooner. Using a real-life example, learn how service virtualization can be applied across the lifecycle to shift integration, functional, and performance testing to the left. Gain an understanding of how service virtualization can be incorporated into your automated build and deployment process, making continuous testing a reality for your organization. Learn what service virtualization can do for you and your stakeholders. The ROI is worth it!TRANSCRIPT
nt Session
Presented by:
Allan Wagner
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
T10 Concurre4/8/2014 2:00 PM
“Continuous Testing Through Service Virtualization”
IBM
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Allan Wagner IBM
IBM-certified IT Specialist Allan (Al) Wagner is currently a technical manager and evangelist with IBM, driving thought leadership, strategic initiatives, and tangible solutions with a specific focus on quality management, test automation, and service virtualization. In his fifteen years of practical IT field experience, Al has assisted, supported, mentored, and enabled teams with their IT application infrastructure, development, implementation, and operations challenges. A frequent conference speaker on software quality principles and techniques, Al also has authored or coauthored the IBM Service Virtualization for Dummies book and numerous papers on quality management and test automation.
1
Continuous Testing
through
Service Virtualization
@Al_Wagner [email protected]
Technical Marketing Manager
IBM
Session T10
Test Techniques
Tuesday, April 8th, 2:00pm
1
Business stakeholder expectations Better – Cheaper - Faster
In the past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seemed to have enough
time.
• Software quality suffered.
Today
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
• Software quality continues to
suffer.
2
2
Business stakeholder expectations Better – Cheaper - Faster
In the past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seems to have enough
time.
• Software quality suffered.
Today
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
• Software quality continues to
suffer.
What are organizations “actually“
doing to deliver higher quality
software to market faster?
3
Adding more Testers is not a sustainable model Labor costs continue to increase
Testers
Testers
Testers
Development
Team
Testers
3
4
Adopting technology only increases testing complexity Test environment setup is harder and more expensive than ever before!
external
users internal
users
Application
Layer
Mainframe ERP
Systems
Custom
Applications Databases
Enterprise Service Bus
Functionality Usability
Performance/Scalability
Reliability Consumability
Before, testers
designed tests
to measure
quality based
on a number
of attributes.
5
hybrid
Network Latency
Security/Privacy
Availability
Cost
Data Integrity
Adopting technology only increases testing complexity Test environment setup is harder and more expensive than ever before!
external
users internal
users
private
Business Partner
Services
public
External 3rd Party
Services
Application
Layer
Mainframe ERP
Systems
Custom
Applications Databases
Enterprise Service Bus
Functionality Usability
Performance/Scalability
Reliability Consumability
Before, testers
designed tests
to measure
quality based
on a number
of attributes.
Now, there are
many new
things testers
also need to
consider.
4
6
Faster and more frequent builds are not enough The transformation to Agile isn’t benefitting Testers
Tester
Setup test environment
Developer
Write & deliver code
Nightly build(s) Compile, unit test, publish
Developer
Developer
Developer
Developer
Developer
“N” nightly builds are piling up
7
Faster and more frequent builds are not enough The transformation to Agile isn’t benefitting Testers
Tester
Setup test environment
Developer
Write & deliver code
Nightly build(s) Compile, unit test, publish
Developer
Developer
Developer
Developer
Developer
“N” days to install & configure
“N” nightly builds are piling up
Quality
Gate BVT
Multiple
Test Environments
FVT
Perf Test
Security Test
SIT
UAT
I just spent 2 weeks setting this up &
the build is no good?
When agility stops at testing, the entire
organization hits the wall
5
8
Bottom line – current approaches are not helping! Today’s testing challenges still remain
Development Velocity Testing Labor Test Lab Complexity
Outsourcing labor is no longer
a sustainable model as global
wages are increasing
Product and application
complexity are driving
test lab cost increases
Test teams can no longer
keep up with agile
development
Heterogeneous Environments
Public Cloud Private Cloud
Data Warehouse Mainframe Enterprise
Service Bus
Directory Identity
File systems
Collaboration
Web/ Internet
Routing Service
Third-party Services Portals
Content Providers
EJB
Shared Services Archives
Business Partners
Messaging Services
Traditional
Testing
Other costs
Testing costs
9
What Testers really need! Validating today’s complex applications
Production-like test environments which are:
realistic & complete
available on demand
inexpensive to setup
Heterogeneous
Environments
Public
Cloud Private
Cloud
Data Warehouse Mainframe Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under Test Routing
Service
Third-party
Services Portals
Content
Providers EJB
Shared
Services Archives
Business
Partners
Messaging
Services
Other costs
Testing costs
Automated tests which are:
able to run unattended validating application integration, functionality, security & performance
capable of validating business flows across multiple channels
durable, maintainable & resilient to change
Orchestration from build to production deployment which is:
reliable
repeatable
traceable
Better
Faster
Cheaper
6
10
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience
Continuous
Monitoring Collaborative
Development
Continuous
Release and
Deployment
Continuous
Customer
Feedback
and
Optimization
Continuous
Business
Planning
Monitor
and Optimize
Develop
and Test
Release
and Deploy
Plan
and Measure
DevOps –
Continuous
innovation,
feedback and
improvements
Enables clients to seize market opportunities
and reduce time to customer feedback
Increasing adoption of DevOps Extending Lean and Agile practices across the entire software value chain
Continuous
Testing
11
All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline Development Testing Staging Production
Business
Owners Customers
7
12
Continuous Deployment and Release
All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate BVT
Dev UT
At least 1/3 of is testing!
13
All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate BVT
Test
scripts
Test
data
Test
configurations
Test
reports
Version
Control
Orchestration & Automation
Dev UT
At least 1/3 of is testing!
8
14
Reality check Which answer would you pick?
What percentage of your time are you spending setting up, maintaining, and
re-cycling test environments?
1. 0% to 20%
2. 21% to 30%
3. 31% to 40%
4. 41% to 50%
5. More than 50%
11%
18%
14%
43%
14%
Responses from GRUC Webcast – October 22nd, 2013
15
Solving the Test Environment dilemma Making production-like test environments a reality
Traditional test environments are both underutilized and insufficient – Environments are expensive: hardware, software, and labor
– Tester and developer time wasted waiting for appropriate environment
Agile test environments require four enabling capabilities
Dynamic Infrastructure
Test Data
Management
Deployment
Automation
Test Environments
DevOps Foundation
Service
Virtualization
9
16
Reality check Which answer would you pick?
How does your organization deal with testing bottlenecks due to unavailable or
missing software and services?
1. Manually write simulation stubs to emulate missing functionality.
2. Use commercial service virtualization solution.
3. Defer testing until later when the code is available to test.
4. De-scope testing of that functionality.
5. None of the above.
9%
9%
56%
6%
19%
Responses from GRUC Webcast – October 22nd, 2013
17
What is Service Virtualization? Making the unavailable available for testing
Virtual components simulate the behavior of a service or application during testing
– System dependencies are a key challenge in setting up test environments
– Unavailable/inaccessible services slow down the testing process
– Costly 3rd party access fees can significantly impact testing availability
Virtual components run on commodity hardware, private cloud, public cloud
Each developer and tester can easily have their own test environment
Developers and testers can continue to use current testing procedures and tools
Streamline creation of production-like environments and negative testing
Heterogeneous Environments
Public Cloud Private Cloud
Data Warehouse Mainframe Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under Test Routing
Service
Third-party
Services Portals
Content
Providers EJB
Shared
Services Archives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components
10
18
Virtual components simulate the “real world” behavior
Category Detail
Simple Hard-coded response returned for given input
Non-deterministic One-of-n hard-coded responses
Data driven Input and/or output data specified in external
data source (Excel, file, database)
Data model, stateful
Input and/or output data kept in data model with
complex relationships. Supports CRUD and
other stateful behavior
Behavioral
Extends model-driven to provide pre-packaged
functionality, e.g. shopping basket, real-time
data feed, trading exchange, order matching
What are virtual components? Simulating application functionality, behavior, & performance
19
Test Environment
Volume Constraints
Volume Constraints
2hr. access window
Maintenance Constraints
Database / JDBC
Server/ Web Services
Mainframe / MQ/CICS
Smart Device / SD-IO
MQ/CICS
Web Services
Smart Devices
JDBC
Virtual Environment
Capture & Model
System Under Test
Listen / Capture
Store
Test Case
Repository
Creating virtual components Listen and capture
11
20
Build virtual components from the design specs
HelloService
Turn specifications into virtual components
Creating virtual components Author from scratch using design specifications
21
Continuous Testing reduces risk
Actual Service/App
Virtual Service/App
Service Virtualization is an enabler for continuous testing
Services, applications, systems are introduced into the continuous integration cycle in a prioritized, controlled fashion.
Controlled integration helps isolate defects for faster resolution.
Service virtualization enables continuous integration testing
Using virtual components Enables testers to continuously test what they want, when they want, how they want
12
22
Avoiding the big bang Control integration
Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database ERP
System
Custom
Application
23
Avoiding the big bang Control integration
Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk.
Service virtualization puts control in the hands of the tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Simulate the behavior and
performance of the
dependent components.
Mainframe Database ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service Virtual
Service
Virtual
Service
Virtual
Service
13
24
Avoiding the big bang Control integration
Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk.
Service virtualization puts control in the hands of the tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Simulate the behavior and
performance of the
dependent components.
Begin end to end testing
earlier and continuously. Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service Virtual
Service
Virtual
Service
Virtual
Service
25
Avoiding the big bang Control integration
Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk.
Service virtualization puts control in the hands of the tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Simulate the behavior and
performance of the
dependent components.
Begin end to end testing
earlier and continuously.
Introduce the real
implementations as they
become available.
Mainframe Database ERP
System
Custom
Application Virtual
Service
14
26
Stand up test environments faster Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database ERP
System
Custom
Application
App Layer
new
27
Stand up test environments faster Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the cost of testing and eliminates delays.
Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database ERP
System
Custom
Application
Virtual
Service
App Layer
new
15
28
Stand up test environments faster Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the cost of testing and eliminates delays.
Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
Quickly stand up test labs using
virtual services to simulate
dependencies which have not
changed.
Testers
Business Partner
Services
3rd Party
Services
Virtual
Service
Virtual
Service
App Layer
Virtual
Service
new
29
Stand up test environments faster Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the cost of testing and eliminates delays.
Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
Quickly stand up test labs using
virtual services to simulate
dependencies which have not
changed.
Test applications end to end with
confidence but at a lower cost and
without delay.
Testers
Business Partner
Services
3rd Party
Services
Virtual
Service
Virtual
Service
App Layer
Virtual
Service
new
16
30
Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database ERP
System
Custom
Application
Testers
31
Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software.
Service virtualization makes the unavailable available for testing purposes.
Create virtual components from
development specifications. Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database ERP
System
Custom
Application
JSON REST WSDL
Virtual
Service Virtual
Service
Virtual
Service
Testers
17
32
Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software.
Service virtualization makes the unavailable available for testing purposes.
Create virtual components from
development specifications.
Record the functionality and
behavior of the application using
virtual services.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database ERP
System
Custom
Application
Virtual
Service Virtual
Service
Virtual
Service
Testers
rec
101101100 101101100
33
Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software.
Service virtualization makes the unavailable available for testing purposes.
Create virtual components from
development specifications.
Record the functionality and
behavior of the application using
virtual services.
Execute automated tests at any
layer in the application using virtual
services to simulate functionality
under development.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database ERP
System
Custom
Application
WSDL
Virtual
Service Virtual
Service
Virtual
Service
Testers
101101100 101101100
18
34
Blocking defects - they won’t slow me down Reducing testing down time
Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck!
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database ERP
System
Custom
Application
App Layer
new
35
Blocking defects - they won’t slow me down Reducing testing down time
Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates bottlenecks and increases productivity.
Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database ERP
System
Custom
Application
App Layer
new Virtual
Service
19
36
Blocking defects - they won’t slow me down Reducing testing down time
Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates bottlenecks and increases productivity.
Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
Continue testing without delay.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database ERP
System
Custom
Application
App Layer
new Virtual
Service
37
Blocking defects - they won’t slow me down Reducing testing down time
Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates bottlenecks and increases productivity.
Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
Continue testing without delay.
Replace the virtual service with
the updated source code without
having to redeploy your entire
application.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database ERP
System
Custom
Application
App Layer
new fixed
20
38
The ROI of Service Virtualization The results are real!
3 Year Benefits:
• Increased project delivery capacity 100% (40 projects to 80 projects)
• Avoided hiring three additional FTE's, saving almost $600k.
• Reduced incidents found in production from 2.5 to .3 per project, saving $3 million
• Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k
• Reduced consulting and third party testing fees with better test automation, saving $408k
• Eliminated the need for additional hardware and labor to set up test environments, saving $700k
• Generated new revenue for the business – estimated at tens of millions!
A major European bank implemented service virtualization and automated integration
testing as part of an Agile transformation to enable continuous testing. As a result of
IBM capabilities, their system integration test (SIT) process was reduced from 3
weeks down to four hours!
The Total Economic Impact™ of IBM Rational Service Virtualization and Test Automation Solutions, a July 2013 commissioned study conducted by Forrester Consulting on behalf of IBM
39
Orchestration
Continuous Testing
InfoSphere Optim
Test Data Management Rational Test
Virtualization
Server
Rational Test
Workbench
• Capture service
behavior and model
virtual services
• Test services and
applications long
before their user
interfaces becomes
available and do
integration testing
• Performance test,
regression test,
functional test &
integration test
• Virtualize
heterogeneous
hardware, software,
and services to
provide 24x7 testing
capabilities
• Reduce the cost of
traditional testing
environments
• Build Virtual Services
to simulate over 70+
protocols
Rational
Performance
Test Server
• Reuse Rational Test
Workbench scripts to
drive performance
testing
• Combine virtual
services and the real
implementations to
performance test
earlier
• Probe for
identification of
system bottlenecks
• Create referentially intact, “right-sized” test databases
• Automate test result comparisons to identify hidden errors
• Protect confidential data used in test, training & development
• Refresh test environments faster shortening iterative testing cycles and accelerating time to market
IBM’s continuous testing solution
Rational Quality
Manager
• Track quality using a
comprehensive and
customizable test
plan
• Link test cases with
requirements,
defects, development
work items and builds
• Author and execute
automated or manual
test scripts
• Provide up-to-date
project quality status
using customizable
dashboards
Test Data
Management Service
Virtualization
Test
Automation
with integration to
Collaborative
Development
& Test
Completing the Continuous Testing story Service Virtualization is the enabler but it is only part of a Continuous Testing solution
21
40
Orchestrate and automate the deployment of applications, middleware
configuration, and database changes into development, test and
production environments.
Databases Internal Messages
Third-party Services
virtual components
Simultaneously
test across
multiple test
stages
Dev QA
Test Data
Management
Deploy
Provision
Test
Combining continuous testing with Orchestration from build to production Reliable, repeatable, & traceable deployment automation
Deployment
Automation integrated with
Test Environments
Dynamic Infrastructure
Test using real world
network conditions
Service
Virtualization
Test
Automation
Deploy and refresh
right size secure data
Deploy what is ready,
virtualize the rest
Continuously test in
production-like env.
Network
Virtualization
41
Continuing on your Service Virtualization journey Increasing awareness and accelerating learning
Get your copy today!
Service Virtualization
For Dummies
ibm.co/ServiceVirtualizationForDummies
Visit
Service Virtualization
on ibm.com
ibm.co/servicevirtualization
22
42
Continuous testing as our planet gets smarter If you think testing is hard now, just wait
intelligent interconnected instrumented
We are already on the road to the Internet of Things
Sooner than later, someone is going to ask you to test it!
43
23
44