confoo: moving web performance testing to the left


o What is the impact of Web Performance?

o Your Web Site and the need for Performance Testing

o Typical Performance Testing

o Issues found in Performance Testing

o Making Performance Testing Continuous

What do users care about?Survey Says:

Source: http://www.soasta.com/blog/web-performance-mothers-day/

Fast:theybuymore.Slow:theybuyless.

PerformanceaffectseverybusinessKPI• Revenue• Conversions• Usersatisfaction• Userretention• Timeonsite

• Pageviews• Bouncerate• Organicsearchtraffic• Brandperception

Impact of a 1-second delayA site that earns $100K per day

could lose $2.5M in salesin one year

…website slowdowns occur 10X more often than outages.

Source: TRAC Research

Free download:soasta.io/timeismoneybook

12

Anatomy of Your Web App

Responsive web site Web Servers

• Desktop Users

• Mobile Users

• Same or similar content

o Adapting to different

screen sizes

Mobile optimized

• “m-dot” site for mobile

• Custom built for mobile devices

All have html, images, JavaScript, css, plus third-party content

13

Native App

• Android – Java

• iOS – Objective C/Swift

• Mobile app framework

o Same source code

Hybrid App

• Mix of native code + html

All have backend calls for login/authentication, account

settings, database calls, transactions, plus third-party SDK

calls

Mobile Apps and Back End ServicesWebSite/Mobile

Back end

14

The Need for Load Testing

• Everyone runs load testing on their website, right?

o Load and Performance testing is often not done or too late

o Black Friday, Cyber Monday, Holiday failures

• What’s the need for mobile web load testing?

o Mobile is growing faster than desktop

o Responsive sites and mobile sites deliver different content to

mobile devices over mobile network

• What about mobile app load testing?

o Mobile app back ends serve different content to mobile apps

o Remember Pokemon Go? It wasn’t the app that failed.

15

When is your Peak Season?• Peak is year-round

o Super Bowl Sunday – 110 million viewers

o Bi-weekly or monthly payroll processing

o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day

o Quarterly processing and filing

o U.S. Tax filing

o Spring, Summer, Fall & Winter season sales

o July 4th, Christmas in July

o Back to School

o Olympics

o Halloween

o Election Day

o Singles Day – world’s biggest shopping day, $9B in 12 hours

o Black Friday, Cyber Monday and Holidays

Back to top to repeat again next year!


s

Memory leaks

Scale of Test

Stag

e / T

eam

Dev & Test

Release& Deploy

Network& Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collectionDatabase thread counts

Inefficient database queriesSlow pages

Conflict with other apps

Test Lab Staging Production (100% +++)

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Tuning for full-scale…and well beyond


Testing In GeneralLinear approaches include delays between testing

UtUnit Testing

FtFunctional

Testing

Development Testing Operations

PtPerformance

Testing

MoMonitoring

Legacy Approach: Silos and waiting

TIME


Eliminating Delays Compresses TimeEnabling more testing to be done earlier and more often

UtUnit Testing

FtFunctional

Testing

Testing

PtPerformance

Testing

MoMonitoring

Legacy Approach: Silos and waiting

Agile Approach: Automation and Continuous Testing

OperationsDevelopment

Dev OpsTesting

TIME


Load Testing with Specific ObjectivesTests at different stages deliver higher coverage

Component scalability

Performance baseline

Integration testing

End to End w/ ServiceVirtualization

Unit scalability

Database testing

3rd party services

Live E2E tests

Stress testing Soak test

(reliability)

Failover

Capacity

Latency

PerformanceRegression (smoke tests)

Network

Testing (staging) Operations (prod)Development (sandbox)


o Small Batch Sizes

o Comprehensive Version Control

o Simple Branch Strategies

o Automation

o Continuous Feedback

o Emphasis on working builds

o Consistent environments

o Developer – Tester Collaboration

21

Shifting Left

• Agile means shorter development cycles

o Used to be months, now weeks

o Quality sign-off within sprint

• Developers and QA working together

o Developers write unit tests; some teams swap roles

• Performance testing is moving to left

o Meet performance SLA in all new code

• Automation with CI tools including Jenkins

o Code that meets quality goals gets promoted automatically

Testing is moving to the left, including unit, functional, and performance

22

Shifting Right

• Performance testing in development

o Development integrating performance tests into SDLC

o Strong adoption of open source test tools, such as JMeter

• Performance engineers using commercial tools

o Execute performance tests at scale, from multiple geos

o Testing not just software but infrastructure

• Testing tools addressing both sets of users

o Adding JMeter support, Continuous Integration automation

Moving performance tests from development to production


• Scalability issues

• Capacity issues

• Concurrency issues

• Reliability issues


• What does a problem look like?

o Longer response times is a clue

o ”High” CPU / Memory / Storage / Network utilization

• Solutions?

o Ops Problem? Can often be “solved” with hardware

o Tall stacks -> Wirth’s Law: “Software is getting slower more rapidly than

hardware gets faster”

• Testing in development

o Test at loads greater than 1 user

o Monitor resource usage across builds and compare to baseline / acceptable



o Response times vary related to load on system

o Increasing CPU / Memory / Network utilization

o Increasing I/O latency

• Solutions?

o Cannot be “solved” with hardware

o Re-architect solution for high-capacity


o Test at loads greater than 1 user, compare as load increases

o Monitor resource usage across builds and compare to baseline / acceptable



o Infrequent functional issues that seem only to occur under load

o Process crashes and restarts

o Not easily reproducible

• Solutions?


o Re-architect solution for high-concurrency


o Test at loads greater than 1 user

o Monitor functional results while testing at increasing loads

o Monitor processes for crashes and restarts



o Process crashes and restarts

o Memory or object leaks

o More frequent garbage collection

o Decaying response times

• Solutions?


o Find resource leaks based on failure points


o Test at loads greater than 1 user for increasing amounts of time

o Monitor processes for crashes and restarts

o Compare behavior and resource usage as time increases


• Additional measurements and aggregation

o Response Time Average / 90th (92/95/98th) Percentile

o Minimum / Maximum Response Times

o Completion / Error Rate %

o Transactions per second

o Size of Response

o Network Throughput

o Measurement against previous run or baseline

o Use more than one of the above!


• The Path to a Performance Culture

Run load tests via

JenkinsDaily (or more frequent) updates keep

teams “performance focused”

Identify

bottlenecks &

inform teams,

just like unit

and functional

tests in

Jenkins

Jenkins launches

suitable cloud-

based test

environment

Manage with a new metric!

30

Test Continuously at Different Load

Test from Development to Integration to Staging to Production

31

JMeter

• Hosted by Apache Software Foundation

o http://jmeter.apache.org

• Designed to load test functional behavior and measure

performance

o Web – HTTP, HTTPS; SOAP / REST; FTP; Database, more

• GUI design

• Plug-ins

• Tests as xml

Popular Open Source performance testing tool


HaveyoumetJenkins?

#1 CI/CD Automation Engine

• Open Source

• Highly Extensible

About 8 years old

Easy to install/use

1000+ plugins

Widely adopted

• 100K+ installations


Over 1000 Jenkins Plugins for Tool Integration

Integrated Development Environments (IDEs), editors/compilers/debuggers, code

review and collaboration tools.

Source code control systems and related asset

management tools.

Automated API-based testing frameworks.

Release management and release automation software

Automated API-based testing frameworks, static code analysis tools.

Load Testing Tools and Security Scanning technology

Automated and manual UAT tools

Release management, release automation, and

change control software.

Environment and Configuration Management SoftwareDevelop

Commit

Build & Integrate

Unit Tests

Release(to Test)

AutomatedRegression Tests

Release(to Prod)

User Acceptance Tests

Load & Security Tests


Jenkins is the Hub of the CD Ecosystem

On-premise or

in the cloud

Plug-ins for all

your tools

RunDeployStageTestBuildCommit


SOASTAJenkinsPlug-in

o OpenSource:onlineatjenkins-ci.org

o AutomaticallyavailableonCloudBeesDEV@cloud

o CompletecontrolovertestautomationfromwithinyourJenkinsbuilds:

o TotallyintegratedintoJenkinsenvironment

o Similarplug-insavailableforHPLoadRunner,BlazeMeter,JMeter,Load.io,etc.

36

Automate Your Testing and Promotion

Use Continuous Integration tools to automate testing

37

Results of Shifting Performance Testing Left

Using Continuous Integration tools to automate testing

• Run incremental performance tests automatically with each iteration

• Establish a trend line for performance early• Developers find code and architecture issues early• Performance teams focus on end-to-end infrastructure

and user related testing• Minimize production escapes and implement

production testing

38

Questions, Resources & Next Steps

• Holiday Readiness

o soasta.com/holiday-readiness

• The Performance Beacono soasta.com/blog

o soasta.com/blog/2015-popular-posts-web-performance

• Webinars

o soasta.com/webinars

o Load Testing for Peak; RUM-Based Testing

• CloudTest Lite

o soasta.com/cloudtest: Free community version for 100 VUs

Resources


How fast should your site be?


Speed costs money…


…and how much load should it handle?


Your customers will be your guide…


RUM

How fast is my site?

What locations?

From where?

Which pages?

Which flows?

Test

Full scale load

From any location

Going through

real user flows

With real-time

analytics

Real-Time Analytics with Real User Measurement


• Tying Business Metrics to (ROI) with website performance metrics and real user data

CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015

Building performance tests that model user behavior on your site

First you need to determine the workload for the test.

How are companies doing that today?

• Guessing.

• Logs.

• 3rd Party tools.

What if I have a new system?


o What information I need?

● When is your peak second/minute/hour/day?● What is your peak volume? (active sessions, unique users, page

visits, orders, ...)● What is the median session length?● What is the distribution of user paths?● What is the distribution of devices?● What is the distribution of locations?

o Good Test Data● Search Terms● Product Distribution● Browse Distribution● Cart Size● Order Size

How to get a realistic load testFactors to consider


Use real user data to model real user tests

ROI of Performance

Focus on the ROI of Performance


Test how you want, when you want, where you want


• Real Users (RUM) provide deep data about your visitors

o No other way to know everything about your users on your site,

from desktop and mobile

• Virtual Users tell you how your site or app will do at scale

o You can’t get a million friends to visit your site or app at the same

time!

JMeter, Load.io, LoadRunner, SOASTA CloudTest let you build and

execute load tests at scales from hundreds to millions of virtual users


• Session paths

o How are your users going through your site

• Mobile App Performance

o How are your mobile apps performing?

o What are users doing in your mobile apps?

• Revenue

o How much money are you making from each user and where?


What was the busiest Day, Hour, Minute, Second?


What were most popular page flows with think times? Which APIs?


• Testing to match realistic situations

o Users are not all from one location on a fast network

o Packets get delayed, lost, re-ordered

• Apps perform differently as network conditions worsen

o Reduced throughput

o Timeouts occur, connections are lost

o Transactions take longer, longer sessions

Adding real network modeling to load and performance testing

Walmart found…

Converted shoppers were served pages that were 2X faster than pages served to

non-converted shoppers.

Non-buyers were served category pages that were 2-3 seconds slower than category pages

served to buyers.

For every 100 milliseconds of improvement, incremental revenue increased by up to 1%.

http://www.soasta.com/blog/mobile-web-performance-monitoring-conversion-rate/

Conversion rate barely decreaseswhen load times for “checkout” pages degrade

Conversion rate shrinks by about 50% when load time for “browse” pages

increases from 1 to 6 seconds

What is the Conversion Impact Score?The Conversion Impact Score (CIS) is a relative score that ranks page groups by their propensity to negatively impact conversions due to high load times. For each page group, the Conversion Impact Score is calculated using the proportion of overall requests that are associated with that group, along with the Spearman Ranked Correlation between its load times and number of conversions. The Conversion Impact Score will always be a number between -1 and 1, though scores much greater than zero should be very rare. The more negative the score, the more detrimental to conversions that high load times for that page group are, relative to the other page groups.

How much impact does the performance of this page have on conversions?

1. Product2. Category3. Shopping cart4. Home5. Search results

http://www.soasta.com/blog/website-monitoring-conversion-impact-score/

69

Test in Production

Some issues will only be found at scale load on production systems

Don’t forget your third party providers: CDNs, external scripts

70

Measure Mobile App Performance Under LoadAll mobile apps perform well when there is no load

71

Now That the Test is Over

• Analyze results for future areas to address

o Weak or stress areas that aren’t likely this time but …

• Identify third parties that were affected by load

o Coordinate with suppliers to address issues

o Align schedules or at least be aware of their changes

Even if the test passed, you’re not done

72

Issues Will Happen

• “I am 100% confident that everything we tested will work

just fine.” – Nordstrom Performance Leader

o When loads are different than what you modeled, may have

problems

o E.g. At Nordstrom, during a recent anniversary sale, mobile

sign-ins were 9x greater than anticipated

o Used RUM to get to the root cause of the issue

Realize that Issues Will Happen Where You Can’t Predict Them

Q & A

Resources

Next Steps

74

CloudTest – Load Testing for mobile web, apps

• Load testing from development to production

o Continuous testing at any scale, on-prem or in the cloud

• Real-time analytics with customizable dashboards

o Safely test in production with full visibility

• Run tests live or automatically

o Full hands-on control or hands-off automation

• Full end-to-end view

o Front-end performance to back-end systems

Fast, scalable, affordable load testing for performance optimization

75

Questions, Resources & Next Steps

• Holiday Readiness

o soasta.com/holiday-readiness

• The Performance Beacono soasta.com/blog

o soasta.com/blog/2015-popular-posts-web-performance

• Webinars

o soasta.com/webinars

o Load Testing for Peak; RUM-Based Testing

• CloudTest Lite

o soasta.com/cloudtest: Free community version for 100 VUs

Resources