Is the current model of load testing broken?

Steve ThairSeriti Consulting@TheOpsMgr

WEB

Some background…. My User Group – Dec 2011 -

http://www.meetup.com/London-Web-Performance-Group/

CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

http://www.slideshare.net/sthair/continuous-integration-a-performance-engineers-journey

Build

Environment

ContinuousLoad

Injection

Production logs

End User Response

Perf Trends

Perf Details

Delivery Team

Usage Profile

For Each Data Point

Alert

Deploy

Service/API response

Measurements

System monitors

Fail Fail to meet

SLA’s?

Deviation from

normal?

Ready for

Test?

08/04/2023

@TheOpsMgr #ukcmg #webperf

The Killer Comment…

“We’ve had to look at separating load injection from performance

measurement”.- Andrew Harding, Betfair

(c) Seriti Consulting, 2011

4

http://www.seriticonsulting.com/blog/2011/12/9/is-the-current-model-of-loadperformance-testing-broken.html

5

@TheOpsMgr #ukcmg #webperf

So why are you paying all that money for that

expensive brand-name load testing tool then?

What are you using to measure

Performance then?

Surely there are open source or

cheap cloud tools if all you want is load?

What are key reasons behind that decision?

What other issues might we

have with “traditional” load testing models in

a Web 2.0 world?

6

@TheOpsMgr #ukcmg #webperf

Defining the “current model”

“Stephen didn’t explicitly define what the current model of load/performance testing is…”

Alex Podelko

http://applicationperformanceengineeringhub.com/is-the-current-model-of-loadperformance-testing-broken/

(1) Waterfall Development Cycle(2) Load Tool compiles the report(3) Reporting at the end of the test(4) Request / Response paradigm(5) Well-defined customer journeys

“Current Model Straw Man”

8

@TheOpsMgr #ukcmg #webperf

Testing in a CI/Agile world Performance testing initiated every time code is committed to

the CM repository (e.g. SVN, GIT etc) Betfair issue was that their system was complex with many

layers of caching etc so it took longer to “warm up” the environment (to achieve a steady performance state) than they had between check-ins…

So they needed “continuous injection” to keep the environment constantly warm…

The test tool never “stopped” to “compile the report”… So any tool that “reported at the end” wasn’t as useful…

9

@TheOpsMgr #ukcmg #webperf

Test tools in a APM & RUM world “traditional test tools” generally have the ability to deploy

agents to gather metrics from the target environment But the depth of analysis and correlation falls well below that

of modern Application Performance Management tools e.g. AppDynamics

APM tools offer a deeper insight and better event correlation across tiers…

And they are getting (much) cheaper…

@TheOpsMgr #ukcmg #webperf

APM Measurement

@TheOpsMgr #ukcmg #webperf

Google Real Time Analytics

12

@TheOpsMgr #ukcmg #webperf

WebSockets HTTP 1.1

Start Timer GET /index.html Response 200 OK

Stop Timer WebSockets

Start Timer Socket “upgrade” N many frames…

Stop??? Bi-directional socket channel

13

@TheOpsMgr #ukcmg #webperf

HTTP 2.0 HTTP 1.1

Sequential ordered

HTTP 2.0 (& SPDY) Multiplexed over a

single connection Responses

returned out of sequence…

Hard to time!

http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy

14

@TheOpsMgr #ukcmg #webperf

Website Complexity 263,000 entry points…

15

@TheOpsMgr #ukcmg #webperf

Visitor Flow How many paths thru a

website? Classic script-driven

approaches can’t hope to address the complexity

Network and log file replay solutions?

16

@TheOpsMgr #ukcmg #webperf

A PCAP Solution to Replay? POC solution based on Cloudmeter

Pion + custom scripts Read a PCAP (network capture) Identify the HTTP traffic Filter it (based on your requirements) Parameterise it (query strings, POST

parameters etc) Randomise inputs from SQL, CSV

etc Replay it against a test environment

i.e. change the base URL Amplify & rate throttle req/sec as

requiredhttp://www.cloudmeter.com/pion/data-processing.php

@TheOpsMgr #ukcmg #webperf

So what’s the Answer? Use the cheapest method to generate load that you can

find… Move away from scripting-based approaches towards using

real-user session replay (if possible!) Generate load continuously Measure continuously

using APM & RUM type tools – FOSS or Commercial Look for changes in histograms, averages, standard dev etc

Protocol and Framework aware instrumentation AFAIK this currently doesn’t exist…

17

Personal Opinion!YMMV…

08/04/2023

(c) Seriti Consulting, 2011

18

@TheOpsMgr #ukcmg #webperf

@LDNWebPerf User Group! Join our London Web Performance Meetup

http://www.meetup.com/London-Web-Performance-Group/ Next Wednesday 17th Oct – 7pm – Central London Follow us on Twitter @LDNWebPerf #LDNWebPerf & #WebPerf

08/04/2023

(c) Seriti Consulting, 2011

19

@TheOpsMgr #ukcmg #webperf

About Me 21yrs IT experience.

Started with www in 1998 (IIS3! Site Server 3!). Web Architect @ BNP Paribas, CSFB etc Web Operations Manager for www.totaljobs.com, www.tes.co.uk Professional Services Manager @ www.siteconfidence.com Seriti Consulting – specialising in web operations, management and

Performance e:stephen.thair@seriticonsulting.com m:+44 7971 815 940 Twitter: http://twitter.com/TheOpsMgr Blog: http://www.seriticonsulting.com/blog/ LinkedIn: http://uk.linkedin.com/in/stephenthair Skype: seriti-steve

08/04/2023

(c) Seriti Consulting, 2011

20

@TheOpsMgr #ukcmg #webperf

Questions?