Performance Testing in New Contexts

•@ericproegler• Performance Engineer• 13 years in (back-end) performance,

20 in software• Speak Easy Mentor (speakeas.ie)• AST Board (associationforsoftwaretesting.org)•WOPR Organizer (performance-workshop.org)

How I Learned to Perf Test…

• Monitor everything: discrete and limited resources (Windows 2000, on Pentium 3)

• Injectors every 50-100 virtual users

• Mercury, Silk, QALoad, Rational Tools

How Is It Holding Up?

• Still lots of projects where we can monitor discrete and limited resources, to some extent

• Injectors every 500-1000 virtual users

• More on tools later, but there have been a lot of changes…

Resources, Shared• Magical SAN thinking

• Virtualization at first: We can watch the Virtualization Host Resources and model?

• Production and the problem of shared resources

Resources, Bundled

• Resources Are Hard to Trace

• Dynamic reallocation: How vSphere works

• Maybe we have to trust the Admin? Or look over their shoulder? Or enlist them?

Resources, Evaluated

• CPU Ready requires host-level knowledge

• The paradox of fewer vCPUs

• Overcommitted resources – the patching story

Resources, Abstracted

• Resources Are Impossible to Trace

• Now we HAVE to trust the Admin

• Even with our injectors…

Resources, Outsourced

• Now there isn’t an admin we can talk to, and it’s not possible to see hardware/virtualization host layer

• Lots of claims about scalability/elasticity, though most systems still engineered with specific numbers of servers, config files

It’s All Out of Our Hands

•We don’t know who the admin is

•We have response time against load models – but no resources. Watch carefully for these to increase.

Story time!

Large Scale Mobile App

• Linked to prime-time TV -- Asks trivia questions during live TV shows; gamers score points they can use to buy stuff; TV show sponsors pay for “sticky viewers”

• Complaints by users not being able to connect and answer questions; seems to occur above 20k concurrent users, but trigger conditions not evident

Large Scale Mobile App

• Growing fast, need to be able to scale to support mega shows

• Applications run on Amazon EC2

• Startup budget; can’t afford commercial tools. Found a little to use a traditional tool just for monitoring

• Want to start with 20k users, eventually run with 50k users

Load Testing in the Cloud• BlazeMeter is Jmeter in the cloud. • Jmeter/Selenium/Gatling/Grinder/etc is

free – like a puppy!

Response Times (milliseconds)

Request/Action #Samples

Average

Median

90% Line Min Max Hit / s KB / s

ALL Requests 1,734,829 24.6 24.5 61.7 2 1038 945 11,949

Get_Settings 20,000 13.6 13.6 16.5 10 45 20 70

Get_event_data 20,000 9.0 8.9 12.9 7 60 20 24

Information_about_the_current_game

20,000 23.6 23.5 29.4 15 78 20 18

LANDING_HTML_PAGE 20,000 11.3 11.1 117.7 5 152 20 36

Query Game Phase

1,112,998 28.1 28.1 48.1 18 667 618 569

Vote 181,831 35.6 35.7 61.3 13 245 151 73

Card Time Votes1 14:13:35.436 14942 14:16:35.665 53273 14:19:35.892 90484 14:22:36.142 125945 14:25:36.423 159976 14:28:36.707 180867 14:31:27.027 180858 14:34:37.366 180809 14:37:37.640 18082

10 14:40:37.961 1808011 14:43:38.256 1

Total 134,874

Actual number of votes in 30

minutes

Expected number of

votes

My Takeaways• Used to the level of information traditional

tools provide; had to figure out how to get it (run transform on several hundred MB of xml, thin data to get to Excel-sized chunks, etc).

• Would have started with a database if we could do it again, Tableau for visualization

• A good tool to add to the toolbox for web sites at larger scales

Story time!

E-Commerce Site• Wants to test with 5,000 virtual users• Number of VUs a little larger than we

were confident we had computation and bandwidth to support (at the planning stages)• Customer not interested in standing up

load generators• Some budget for tools• Ended up being a config (CDN) and ops

(DR site) test project

Injection in the Cloud, First Hand• NeoLoad – a tool we already knew:–Affordable compared to other commercial

tools–Great analysis engine–Credits system, rental terms–Medium and Large Injectors,

US/EU/Asia/etc

• Called 5,000 Users From Geographically Distributed Engines– Incoming Bandwidth can be an issue…

My Takeaways

• Was actually pretty seamless, analysis was as easy as ever

• Buy plenty of credits, so you don’t have to ration them

What I’m Doing Lately

• Always Cloud Injectors

• Geographic Diversity is table stakes, and necessary for complex sites (Users, CDN…)

What’s Next?• Automation Squared -> Automation

Cubed

• An API Manifest converted to tests

• Micro-Perf Tests, every build, or constantly

• What do you think the future looks like?

Thanks!

@ericproeglerassociationforsoftwaretesting.org

performance-workshop.org

.com

See you next time!