agile load testing in the real world

© Utopia Solutions

Agile Load Testing in the Real World

Lee Barnes, CTO Utopia Solutions @USI_LeeBarnes

© Utopia Solutions 2

Key Questions

What needs to change?

How do we make those changes?

Why don’t traditional performance testing models fit nicely with agile?

© Utopia Solutions 3

Case Study

Background

• Large global manufacturer • Performance Testing CoE in operation for ~10 yrs • Seen as a huge value to the enterprise • Began moving toward agile a few years ago…

and the luster of the CoE began to dull…

© Utopia Solutions 4

Traditional Performance Testing Model

Center of Excellence

Project 1

Project 2 Project N

Provides Consistency in… • Resources / Skills • Processes • Deliverable Quality • Technology

Request Test

Project Team

Center of Excellence

Perform Discovery

Develop & Execute

Deliver Results Evaluate Results

Info / Environment

© Utopia Solutions 5

• One size fits all • Strict entry criteria • Performed late in SDLC • One-way communication of

results

Service Characteristics

CoE in the Waterfall Years

Plan Design Build Test Deploy

Performance Testing

Too late!

© Utopia Solutions 6

Agile Development

Analyze

Develop

Integrate

Acceptance Test

Deploy

Where does performance testing fit?

© Utopia Solutions 7

Early Attempts on Agile Projects

Sprint 1 Sprint 2 Sprint 3 … Deploy

Agile

Performance Testing

Same Process…

Same Risk!

© Utopia Solutions 8

Early Attempts on Agile Projects

Sprint 1 Sprint 2 Sprint 3 … Deploy

Agile

Performance Testing

Performance Testing

Performance Testing

Performance Testing

CoE Approach • Rigid • 4 week test cycle

Agile Approach • Flexible • 2 week sprints

+ =

© Utopia Solutions

Back to the drawing board…

© Utopia Solutions 10

Early Cycle / Ongoing

Performance Analysis

Establish Objectives

Ultimate Goal…

Late Cycle Performance

Testing Shift Left

© Utopia Solutions 11

Identify Gaps

What’s stopping us from analyzing performance continuously?

• Performance objectives weren’t part of user stories

• Agile teams lacked performance engineering expertise

• Performance testing knowledge and tools were locked in the CoE

• APM tools / skills were non-existent

© Utopia Solutions 12

Disclaimers

It was not easy.

The changes discussed on the next few slides were implemented over a two year duration.

It was not painless.

© Utopia Solutions 13

Performance Testing in an Agile Environment

Build performance objectives in to user stories

Make performance everyone’s job

Create a role to coordinate performance activities

CoE performance test architects become mentors

Begin developing a performance focused culture

© Utopia Solutions 14

Performance Testing in an Agile Environment

Map business and end user requirements onto system components

Create both single and multi user tests

Benchmark in development / QA environment

Look for trends in measurements vs. absolutes

Use service virtualization to stub out 3rd party or yet to be developed components

Execute unit performance tests as part of continuous integration

Implement Unit / Component Performance Testing

© Utopia Solutions 15

Waterfall Agile

Responsibility for Performance

Performance CoE

Role of CoE End of cycle PT

Role of development

Throw system over wall Wait for results

Tools Traditional load test tools

Environment Dedicated / production-like Deliverable Boiler plate results delivered

just prior to release

Waterfall Agile

Responsibility for Performance

Performance CoE Everyone

Role of CoE End of cycle PT Provide mentorship to agile team Pre-deployment system PT

Role of development

Throw system over wall Wait for results

Continuous focus on performance

Tools Traditional load test tools Traditional load test tools APM tools Service virtualization Continuous integration

Environment Dedicated / production-like Development / desktop Deliverable Boiler plate results delivered

just prior to release Continuous feed of actionable data

Performance Testing in an Agile Environment

Transformation Summary

© Utopia Solutions 16

Summary

Key takeaways… • Agile performance testing is not more

/ quicker performance tests • Success requires an shift in people,

process and technology

© Utopia Solutions 17

Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: lee.barnes@utopiasolutions.com Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com

?