enabling agility with continuous integration testing
DESCRIPTION
Monica Luke, IT symposiumTRANSCRIPT
Enabling Agility with Continuous
Integration Testing
Monica Luke September 18, 2012
WELCOME
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract. The development, release, and
timing of any future features or functionality described for our products remains at our sole
discretion.
Performance is based on measurements and projections using standard IBM benchmarks
in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.
Agenda
• Integration Testing Challenges
• Continuous Integration Testing
• Agile Integration Testing
Taking a new look at integration testing
.94
.92
.98
x3
.97
Given high
individual
systems’
reliability
(>.92)
Unreliability
increases
exponentially:
R = .79
Do you know
what’s
underneath the
surface?
“In practice, integration has proven to be the most enlightening activity in
exposing the risks and uncertainties in software development.”**
**Walker Royce, Chief Software Economist at IBM
4
Balancing
Quality and Speed
Increasing
Cost of Quality
Increasing
Development Complexity
An estimated 60 - 80 percent of the cost of software development is in rework*
* Source:http://www.sei.cmu.edu/about/message/
Outsourcing labor is no longer
a sustainable model as global wages
are increasing
Product and application
complexity and size
are increasing
Productivity is inhibited as test
teams can no longer keep up with
agile development
Heterogeneous Environments
Public Cloud Private Cloud
Data Warehouse Mainframe Enterprise
Service Bus
Directory Identity
File systems
Collaboration
Web/ Internet
Routing Service
Third-party Services
Portals
Content Providers
EJB
Shared Services Archives
Business Partners
Messaging Services
Traditional
Testing
Other costs
Testing costs
In-house Development
Service-oriented Architecture
(SOA)
Cloud / SaaS End-user Web Application
Mobile Device Access
Third-party IP
Partner Code
Commercial Off-the-shelf
Directory Service
Outsourced and Globally Distributed
Open Source
Main Frame
Heterogeneous systems are complex
And… hard to configure for integration tests
Data Warehouse Mainframe
Directory Identity
File systems
Collaboration
Web/ Internet
Routing Service
Third-party Services
Portals
Content Providers
Shared Services
EJB
Archives
Business Partners
Messaging Services
Enterprise Service Bus
Application
Does your organization look like this?
Tester
Setup integration test environment
Developer
Developer
Developer
Developer
Developer
Developer
N days to install & configure
I just spent 2 weeks setting this up &
the build is no good?$%^??
N nightly builds are piling up
What if we could change it? Takes up to 50% of test cycles
Test Setup is minutes Time spent on meaningful work expands
Package ready to release
Iteration 1… n
Days Days Days Days
Nightly build
Fix Defects Run Test Find Defects
Develop
Agile
QA
Independent Test
Stage
Production
Check ins
Deliver App Package into Production
Test & Verify
Integration Test
Test & Verify
Delay
Manual setup of Test Env
Delay
Manual setup of Test Env
Delay
Manual setup of Test Env
Delay
Manual setup of Test Env
Agenda
• Integration Testing Challenges
• Continuous Integration Testing
• Agile Integration Testing
11
What is Service Virtualization?
•Virtual Services simulate the behavior of an entire application or system
during testing
•Virtual Services can run on commodity hardware, private cloud, public
cloud
•Each developer, tester can easily have their own test environment
•Developer and testers continue to use their testing tools (Manual, Web
performance, UI test automation)
Cap
ture
& M
od
el
Heterogeneous Environments
Public
Cloud Private Cloud
Data Warehouse
Mainframe Enterprise
Service Bus
Directory
Identity
File systems
Collaboration
App Under Test Routing Service
Third-party
Services Portals
Content Providers
EJB
Shared Services Archives
Business Partners
Messaging Services
Databases Mainframe applications
App Under Test
Third-party Services
Packaged apps, messaging services, etc.
Virtual Services
Incremental Integration Testing
Actual Service/App
Virtual Service/App
Services, applications, systems are
introduced into the continuous integration
cycle in a prioritized, controlled fashion
Testing integrations as an evolution
12
Continuous integration Testing Example
C1 C2 C3 ERP WSDL 3rd party UDDI Pass/Fail
Real V V V Test my
own piece
• Test C1 with three virtualized services.
• Can use simple or complex integration scenarios.
• Quick to setup and low-cost.
13
Time
Real V V V R
Real V V V V
Real V V V V V V
Real V V V V R R
Test my
own piece
Integrate
w/another
This won’t
slow me down!
First Complete
System Test
All components
Integrated!
• Integrate C2 and C3… with virtualized backend systems
• Use enterprise integration scenarios for Testing, inc BVT!
• Quick to setup and low-cost.
14
Continuous integration Testing Example
C1 C2 C3 ERP WSDL 3rd party UDDI Pass/Fail
Real V V V
Pass/Fail Time
Real V V V
Real V V V R
Real V V V V
Real V V V V V V
Real V V V V R R
Real R R V V R R
Real R R R R R R
Test my
own piece
Integrate
w/another
This won’t
slow me down!
First Complete
System Test
All components –
Integrated!
The expensive
backend
All Systems UP!
• Gradually integrate with expensive backend
systems.
• Minimize use of backend systems for testing
purposes
• Process offers easy and cost effective defect
isolation
C1 C2 C3 ERP WSDL 3rd party UDDI
15
Continuous integration Testing Example
Continuous Integration testing for builds
Green” builds have
passed meaningful
integration tests
Compile Unit Test Install & Configure
Integration Test
Environment
Publish Results Run Tests
Rational Quality
Manager
Agenda
• Integration Testing Challenges
• Continuous Integration Testing
• Agile Integration Testing
Complex systems Staged agility
Iteration 1 Design, Code, Test Weeks 2 - 6
Integration test
iteration #1
Iteration planning
Week 1
Iteration 2 Design, Code, Test Weeks 8 -12
Integration test
iteration #2
“Stable, working code”?
• Are integration tests pushed later and later in the release cycle?
• Collaboration challenge: Expectation is now that test is aligned with
development
• Are developers doing integration validations?
• Are you compromising on the
definition of stable, working code?
End-to-end Lifecycle Management
RM DM SCM Build Stage QM Monitor
Capabili
ties
CM
Planning
Design Develop Stage Production
Life
cycle
A
cce
lera
tors
Agility@Scale Continuous Delivery
Agile Agile
with Service Virtualization
• Whole team collaboration
• Dev & Test are working on the same code base at the same time
• Change the perspective on team metrics
• Time to fix defects vs number of defects
• Integration testing is happening on every milestone and build
• Whole system testing and verification
• In Agile terms, you are really “done, done, done” on every milestone
• Alignment builds trust delivering on agile vision of self-directed teams
Whole team alignment
22
Rational Test Workbench
Enable functional, performance and integration testing
throughout your project lifecycle
With a scriptless, wizard-driven test authoring
environment and support for more than 70 technologies
and protocols
Rational Test Virtualization Server
Model real system behavior to eliminate test
dependencies and simulate production,
Accelerate testing and reduce the setup and
infrastructure costs of traditional testing environments.
Rational Performance Test Server
Reuse your functional integration tests for performance
testing
Combine your tests to model real-world scenarios and
assess the impact of load on your integrated
infrastructure.
Test
Workbench
Functional Testing
Integration Testing
Performance Testing
Regression Testing
Test
Servers
Load Agents
Virtualization Agents
QA staff
Test Lab
Infrastructure
Introducing IBM Rational’s Service Virtualization and
Test Automation Solutions
23
IBM Rational Quality Management Offerings
Storage
Collaboration
Search & Query Discovery
Administration: Users, projects, process
Presentation: Mashups
Best Practice Processes
Manage
Test Lab
Create
Plan
Build
Tests
Report
Results
Execute
Tests
IBM Collaborative Application Lifecycle Management
Test Management
Rational Quality Manager
Quality Dashboard
Requirements Management Defect
Management
Open Lifecycle Service Integrations
Performance Testing
Functional Testing
Unit Testing Test Data
Management Integration Testing
Test Virtualization
Rational Integration
Tester Rational
Functional Tester Rational
Performance Tester
Rational Test
Virtualization
Optim
TDM
www.ibm.com
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
www.ibm.com