performance testing in the agile enterprise

www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile

Performance Testing in the Agile Enterprise

Chief Technologist, PerfTestPlus, Inc. [email protected] www.perftestplus.com

Co-Founder: Workshop On Performance and Reliability

www.performance-workshop.org

by: Scott Barber

Co-Author: Author: Contributing Author:

Books: www.perftestplus.com/pubs About me: about.me/scott.barber

mailto:[email protected]

http://www.perftestplus.com/

http://www.performance-workshop.org/

http://www.codeplex.com/PerfTestingGuide









http://www.perftestplus.com/pubs

http://about.me/scott.barber

http://about.me/scott.barber


Have you ever wondered…

teams have no idea where performance “fits” in an Agile

development lifecycle?


Because performance

“fit” in an Agile development lifecycle…


Performance is (or should be)

in an Agile development lifecycle!


Overview

Performance Testing is inherently Agile, but...

Integrating Performance Testing

into Agile Efforts effectively is not inherently easy!


Agility in Performance Testing Performance Testing is a repeating cycle of:

Complicated by unknowns, estimations, &

approximations


Agility in Software Development Agile is a repeating cycle of:

Gated by inexact and

variable notions of acceptance


Agile Perf Testing in Agile Dev Putting them together is…

…a tad complicated


Conceptual Models “On Demand”

• A.K.A. Center of Excellence • Call ‘em when you (think you) need ‘em • “Pay” only for what you (think you) need

“On Retainer”

• Performance Tester on team, but mostly independent • Always available

“Full Immersion” • Performance is everyone’s job • Coordinated by embedded Performance Test Specialist • Spans entire lifecycle


On Demand Roles & Interactions:

• Similar to outsourcing except that… • you expect familiarity with the project/product (unlikely) • you expect familiarity with the process (less likely) • you won’t get 100% focus on your project • the assigned resource(s) also report to someone who can

trump the project manager • Minimal interaction with team • In general, they collect info, run tests, & report results.

Strengths: • Fewer performance testers can service more projects • Pooled tools and environments can save time and money • Performance testers benefit from peer collaboration


Performance Test Cycle; On Demand


On Demand Challenges:

• Service tends to be “generic” • Skills/tools/environments may not work for *your* project • Availability often restricted to highest priority projects • Performance testing tends to be both late and infrequent • Validation of tuning often has to wait until the next “on

demand cycle” • No continuity release to release • A core team member must serve as performance lead • Core team responsible for performance exploration

∆ from non-Agile efforts: • Difficult to schedule • System under test a faster moving target • Testing can’t keep up with continuous production delivery


On Retainer Roles & Interactions:

• Performance testing specialist • is embedded into the team full time • may be managed by Project Manager, Product Manager,

or Sr. Engineer • Performance is not (yet) integral to project team culture

Strengths: • Performance testing specialist

• knows project/product/culture/environment details • customizes service to those details • builds relationships that enhance “performance culture”

• Always available • Many more tests conducted than On Demand • Performance exploration and tuning validation “in scope”


Performance Test Cycle; On Retainer


On Retainer Challenges:

• Performance tester(s), tool(s), and environment(s) needed for every project

• Performance testers • Are often seen as “outsiders” • Tend to loose peer collaboration • Might not be fully utilized

• Performance assessments still tend to occur at end of sprints

• No additional support during “crunch time” ∆ from non-Agile efforts:

• Planning is necessarily more reactive • Tool and environment sharing virtually impossible • Testing done in smaller segments


Full Immersion Roles & Interactions:

• Performance specialist coordinates performance related activities including:

• Creation of performance stories • Code profiling and performance unit testing • Performance assessment • Collaboration with production support/capacity planning

• Performance is part of everyone’s job, every day Strengths:

• Performance is, or is becoming, an integral part of the team culture

• No delays, performance testing is done in line with dev • Cross-functional & cross-departmental collaboration • Best chance of no performance nightmares


Performance Test Cycle; Full Immersion

See role and task details in additional resource slides included at the end of this deck


Full Immersion Challenges:

• Performance tester(s), tool(s), and environment(s) needed for every project

• Organizational culture shift typically required • Demands an “architect level” performance specialist • Getting team dynamics “right” takes time – often 2-3

projects ∆ from non-Agile efforts:

• Performance testing at many application levels, daily • Root cause analysis dramatically easier • Tuning is continual • Unity of purpose and goals is essential • The only “external” gates are acceptance testing and

executive business decisions


First Steps Involve management in evolving performance culture Get performance into user stories

• Response times • Resource allocations

Involve developers • Code profiling • Integrate performance measurement into unit tests

Make performance part of story acceptance • Quantitative • Qualitative

Involve performance testers • Throughout the cycle • Gain respect as advisors as well as testers


Valuable Resources Performance Testing Guidance for Web Applications

• Microsoft patterns&practices 2009 • *especially* Chapters 1-7 • Free from http://perftestingguide.codeplex.com/

How to Reduce the Cost of Testing • CRC Press, 2011 • Chapter 16 (free from PerfTestPlus.com) • Order book on Amazon

Other Articles and Websites • An Explanation of Performance Testing on an Agile

Team (part 1, part 2) • What is performance testing? • Software performance testing: There is no 'I' in 'team'

http://perftestingguide.codeplex.com/

http://www.perftestplus.com/Rightsizing_the_Cost_of_Testing.pdf

http://www.amazon.com/gp/product/1439861552/ref=as_li_tf_il?ie=UTF8&tag=perftestplus-20&linkCode=as2&camp=217145&creative=399373&creativeASIN=1439861552

http://www.logigear.com/newsletter/explanation_of_performance_testing_on_an_agile_team-part-1.asp

http://www.logigear.com/newsletter/explanation_of_performance_testing_on_an_agile_team-part-2.asp

http://searchsoftwarequality.techtarget.com/tip/What-is-performance-testing

http://searchsoftwarequality.techtarget.com/tip/Software-performance-testing-There-is-no-I-in-team


Questions


Contact Info

Scott Barber Chief Technologist

PerfTestPlus, Inc E-mail: [email protected]

Web Site: www.PerfTestPlus.com


Oversee, Review and/or

Approve

• Executives set the tone • Articulate performance goals and risks to the business • The rest of the team must

• Achieve goals • Mitigate risks • Assess & report both -- continually

Roles & Tasks (Reference)


Build Capacity Model for n+2 Design n+2Architect &

Model n+2Supervise,

Troubleshoot, Validate and/

or Update Architecture

Model

Observe, Advise,

Troubleshoot, Validate and/

or UpdateArchitecture

Model

Cross-Validate n+2

Capacity Predictions, Architecture

Model Predictions,

Perf Test Results in Staging

• Design to achieve goals & mitigate risk • Establish assessment targets, goals & budgets • Set & enforce sound performance practices • Troubleshoot, tune, update design & refactor



Baseline n+1 in Prod

Build Capacity Model for n+2

Reverse Cross-Validate Release

Readiness Perf Tests in Staging

Validate n+2

Capacity Predictions

in Prod

Cross-Validate n+2

Capacity Predictions, Architecture

Model Predictions,


Provide Current Production Info & Models, Consult When Needed

• Be “hardware architects” • Design to achieve goals & mitigate risk • Establish assessment targets, goals & budgets • Set & enforce sound performance practices • Provide production feedback • Troubleshoot, tune, update design & refactor



Add Performance & Profiling Stories

Develop Stories; Profile; Unit

Level Performance

Testing

Design n+2

• Use coding practices that support good performance • Code to achieve goals, targets, budgets & mitigate risk • Assess components continually • Troubleshoot, tune, & refactor




Reverse Cross-Validate Release

Readiness Perf Tests in Staging

Project Investigative Performance

Testing

SystemInvestigative Performance

TestingCross-

Validate n+2 Capacity

Predictions, Architecture

Model Predictions,


Observe, Consult and

Assist Where

Appropriate

Update Project Performance

Strategy; Design/Develop

Investigative Performance

Tests

Prod Readiness

Perf Testing (System)

• Manage whole system assessment • Own simulation (usage) model(s) • Contribute to component assessments • Cross-validate across environments

performance testing in the agile enterprise

Technology

budgets set

performance test cycle

performance testing specialist

achieve goals

update design

agile efforts

performance testing

goals