ing direct: how we do performance testing in agile environments and bridge the gaps between it teams
DESCRIPTION
Where does performance testing fit within your agile application lifecycle management process? How do you confront the challenges of agile for performance testing and iterative performance regression testing—like staffing, defining non-functional acceptance criteria, environmental constraints, and tool selection? This interactive session, led by a scrum practitioner, will give you insights into these challenges. We will cover topics such as where performance testing fits into agile development, how to get continuous feedback to teams, establishing integrated best practices across business analysts, development, QA and infrastructure teams, and simple processes and tools for quickly delivering quality to your customers. We will also discuss how we bridge the gap between testing and monitoring, and the relationship with capacity planning, disaster recovery, continuity, and system performance validation.TRANSCRIPT
1 ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
ING Direct: How we do performance testing in agile environments and bridge the gaps between IT teams
Todd DeCapua
Production Quality & Process Group Leader, ING DIRECT
Do not put content
on the brand
Show of Hands: Who Is Running Agile?
2
Do not put content
on the brand
Show of Hands: Do You Ever Worry About This?
3
Do not put content
on the brand
Show of Hands: Are You a Performance Engineer?
4
Do not put content
on the brand
Where does performance testing fit within your Agile
application lifecycle management process?
5
Do not put content
on the brand
Philosophical Turning Point:“Flipping The Triangle”
• Agile 2008 – Toronto
• Session Title:
• Flipping the Automated Testing Triangle: The Upshot
• Presented by Patrick Wilson Welsh
• Focus:
• You need three automated testing initiatives, one for each of the three kinds of tests.
• These are: Unit, Integration, and GUI.
• Where should you be investing your time & money.
• Need to flip from the traditional approach.
• Philosophical Turning Point For Me:
• This applies also to performance testing, as we are proving the architecture & design works, so why not flip the triangle of performance testing initiatives; especially with Agile, as we are running bottom up more.
6
http://patrickwilsonwelsh.com/wp-content/uploads/2008/08/flipping-the-triangle.pdf
Do not put content
on the brand
7
Do not put content
on the brand
8
Do not put content
on the brand
It Fits Throughout the Process
9
Do not put content
on the brand
Key Barriers of Agile Performance Testing:
People
(dedicated performance test engineers)
10
Do not put content
on the brand
• Embedded Performance in All Teams at All Roles
• Unit = Developer Role
• Integration = Developer & QA & BA Role
• Functional = QA & BA Role
11
Do not put content
on the brand
Key Barriers of Agile Performance Testing:
Non-Functional Acceptance Criteria
(included in each story)
12
Do not put content
on the brand
User Stories: Mike Cohn – Simple Template
• “As a <type of user>, I want <some goal> so that <some reason>.”
13
Do not put content
on the brand
14
Breakdown of Epic to Tasks with Acceptance Criteria
• Epic (A very large idea or goal)
• Story (One of many work concepts to achieve an Epic)
• Acceptance Test Criteria (Defined objectives to consider a Story done)
• Tasks (One of many specific work items to complete a Story)
So, how does this apply to Performance Testing within Agile?
“As a Father, I want a vehicle that can go forward, left, right, and backward so that I can safely get my family places.”
Front BackSTORY ACCEPTANCE TEST CRITERIA
•Must be able to carry 2 adults, and 2 children at the same time. •Must go at least 60 mph in any direction.•Must have front & side airbags.•I would like to know max speed with max payload.
•Beige exterior with white leather interior. •Four seats.•Four tires with a spare.
Do not put content
on the brand
15
Feature “Doneness”
The code is included in the proper branch in the source code control system.
The code compiles from a clean checkout without errors using production branch [ proposed: and is part of the AHP Build Life which is finally tested in QA. ]
The code is appropriately covered with unit tests and all tests are passing using the production branch.
The code has been peer reviewed by another developer.
Database changes have been reviewed and approved by a DBA.
The code has passed integration, regression, stress, and load testing.
Application Support is aware of the backlog item and the system impacts.
Deployment and rollback instructions are defined, tested and documented.
Until all of these are true, the feature is just unfinished inventory.
Do not put content
on the brand
Key Barriers of Agile Performance Testing:
Environment(s)
(Enabling proper iterative testing with data and scale)
16
Do not put content
on the brand
1717
How: Environments and Quality Gates
DEV IT QA Pre-Prod ProdBVT
VIRTUAL ENV PHYSICAL ENV
Development
Standalone Env for
Build Verification
Verify
App Deployment
DB Deployment
BVT Tests
Development
Integrated Env
Execution of
Smoke Tests
QA Integrated
Env
Execution of
Worklflow Tests
Execution of
Regression Tests
Execution of
Functional Tests
QA Staging Env
Execution
Workflow Tests
Regression Tests
Performance Tests
Release coordination
Production
100% Successful
Deployment
100% Success Build
Verification Tests (or
acceptable workarounds)
XY% JUNIT Pass
XY% Code Coverage
JUnitPerf
QUALITY
GATES XY% Smoke Tests Pass
(or acceptable
workarounds)
JMeter
XY% Regression Tests
Pass
XY% Workflow Tests
Pass
XY% Functional Tests
Pass
LoadRunner
Performance Tests Pass
XY% Regression Tests
Pass
XY% Workflow Tests
Pass
Acceptable Performance
DEV
DESKTOP
Do not put content
on the brand
18
Production
Environment
Development
Environment
Prod(n)
Prod(n+1)
PreProd
Stand InMode
Live-Live
Pre Production
Environment
QABuild
Dev QA
Technology
Infrastructure
•Mock Deployment Tests
•System Performance Validation
•Capacity Analysis & Planning
•Integrated Release Testing
•Infrastructure & Technology Build, Dev, QA
•Manual & Automated Functional Tests
•Unit / Integration / GUI
•Automated Regression Tests
•Performance / Security / Error Tests
•Quality Metrics: Daily & Trend Reporting
•Quality Gate: Coverage % and Pass %
•Governed & Enforced
•Desktop Lab Environment & Tests
•Customer Experience
•Continuous Availability
•No Deployment or Failover Downtime
•Production Quality
•Increased Capacity & Performance
•Safety Net of Stand-In Mode
•Ensure Disaster Recovery Capabilities
•Live-Live Redundancy
•Reduced Risk
Do not put content
on the brand
Key Barriers of Agile Performance Testing:
Tools
(Simple, quick, and continuous feedback)
19
Do not put content
on the brand
20
How: Simple Tool Integration to Support Process
Software Delivery Test Management Auto Build, Test & Deploy
by
-QuickTest Professional
-LoadRunner / Diagnostics
-Shunra
-JUnit / NUnit
-FitNesse
-Axure
-JUnitPerf / LoadRunner /
Service Test / HP Diag Profiler
-Jmeter / LoadRunner /
ServiceTest
Backlog Management
Story Maturity / Estimation
Impediment Management
Sprint Planning
Acceptance Criteria
People
Pro
cess
Tools
Business
BA / Dev / QA
Infrastructure
Architecture
Business / UAT
BA / Dev / QA
Configuration Management
Continuous Integration
Requirements (sometimes)
Test Plan / Test Lab / Defect
Automation / Performance Execution
Analysis / Reporting
Build / Merge / Deploy
Automation Unit Test /
Integration Test
Do not put content
on the brand
Show of Hands: What Do You Use?
GUI
Integration
Unit
21
LoadRunner
Jmeter / Service Test
LoadRunner
JUnitPerf / Service Test /LoadRunner
Do not put content
on the brand
How we bridge the gap between testing, monitoring, and the relationship with Capacity
Planning / Disaster Recovery & Continuity / System
Performance Validation
22
Do not put content
on the brand
Monitoring: How It Works Together
23
Production Operational
“Real Time”
Capacity Planning
“Trending Current”
Performance Lab
“Real Time”
Capacity Planning
“Projection Future”
Improved
Performance
Scenarios
Increased
Production
Quality
Do not put content
on the brand
Prod / DRPre ProdDev / QA
Development
Release Management
Continuity / DR
Capacity Planning
QualitySystem
Performance Validation
Production Quality & Process
Do not put content
on the brand
25
Key Learnings: Performance Testing in Agile
• QA Environment Management
• Quick Refresh
• VM Templates
• Test Data Management
• Root Data Concept
• Built Up & Torn Down within Scripts
• Flip The Triangle
• More on Unit, then Integration, and last on Functional GUI
• Embedded Performance in All Teams at All Roles
• Unit = Developer Role
• Integration = Developer & QA & BA Role
• Functional = QA & BA Role
• Real-time push & pull reporting
• Shift behavior to Quality – Commitment – Continuous Improvement
• Establish Framework & Enable Team To Performance / Non-Functional
Do not put content
on the brand
26
Questions?
Todd DeCapua
Production Quality & Process Group Leader
ING DIRECT
1 S. Orange St., Wilmington, DE, 19801
ph - 302.255.3537
www.INGDIRECT.com | 1-800 ING DIRECT
27 ©2010 Hewlett-Packard Development Company, L.P.
To learn more on this topic, and to connect with your peers after
the conference, visit the HP Software Solutions Community:
www.hp.com/go/swcommunity
28