achieving peak user experiences & optimizing web performance - load testing, stress testing...
DESCRIPTION
Ensuring great experiences for today’s rich and complex web applications requires a new approach to performance testing. Application owners, e-business and marketing managers, QA & IT operations all need to ensure mission & business critical transactions perform flawlessly when: * Releasing new web applications and features * Upgrading or virtualizing IT infrastructure * Adding third-party or cloud-based services such as CDNs, shopping carts and news feeds * Penetrating new markets Join us as Forrester’s Senior Analyst, Mike Gualtieri, and Gomez’s CTO, Imad Mouline, discuss: * Best practices for launching new applications and infrastructure with confidence * Finding and fixing problems prior to launch across the entire web application delivery chain * Maximizing the effectiveness of major marketing campaigns * Why the new “load testing 2.0” approach finds problems legacy tools couldn’t detect, and makes solutions accessible to a broader range of departments and organizations due to their ease of use and on-demand deliveryTRANSCRIPT
![Page 1: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/1.jpg)
![Page 2: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/2.jpg)
Perform Realistic Web Performance Testing ToPerformance Testing To Achieve Peak User ExperiencesExperiencesMike GualtieriSenior Analyst
May 13, 2009
yForrester Research
![Page 3: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/3.jpg)
Realistic web ea st c ebperformance testing is
critical to achieving c t ca to ac e gflawless user experiences.p
![Page 4: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/4.jpg)
Agenda
• Why your customers won’t wait
Wh t i li ti f t ti• What is realistic performance testing
• Recommendations
4 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 5: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/5.jpg)
Agenda
• Why your customers won’t waitWh t i li ti f t ti• What is realistic performance testing
• Recommendations
5 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 6: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/6.jpg)
Performance means:• Raw page load speed Gomez Media Home
Page BenchmarkApril 10 ’09 – May 10 ’09
0:00 - 24:00 EST
6 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 7: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/7.jpg)
Gomez Media ArticlePerformance means:
Gomez Media Article Search Transaction
BenchmarkApril 10 ’09 – May 10 ’09
0:00 24:00 EST
• Customer workflow speed1
0:00 - 24:00 EST
2
33
7 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 8: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/8.jpg)
Realistic Web Performance Testing Is A F t B t P tiForrester Best Practice• Best Practices: Attaining And Maintaining Blazing Fast Web Sites
• Best Practices: Blazing Fast Web Site Infrastructure
• Case Study: How Orbitz.com Takes You Places Faster
• Case Study: How Msnbc.com Is First In News
8 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 9: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/9.jpg)
Blazing FastWeb Sites = CustomersWeb Sites Customers
![Page 10: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/10.jpg)
![Page 11: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/11.jpg)
Traditional web performance testing d ’t t itdoesn’t cut it• Traditional testing is expensive
• Cost of licenses, hardware and training
• Time consuming to implement
• Idiosyncrasies are often difficult to master
• Traditional testing is not realistic• Scope is limited to internal clean-room testing
• Not focused on end-user point of viewp
• Misses too many pieces in the web application delivery chain such as CDN, 3rd-party components,
11 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
geography, etc…
![Page 12: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/12.jpg)
Most Developers Are Building Web Apps.“What types of new custom applications are your firm’s developers building”
(multiple responses accepted)
12 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
2008 Base: 977 platform software decision-makers at North American and European enterprises and SMBs
![Page 13: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/13.jpg)
New trends make optimal user i i t texperiences even more important
• The Millennial generation expects more– 18 to 27 year olds– The tech-savvy generation
• Web site redesigns– RIA
• More Web sites are composed of 3rd-party componentsVi t li ti f i f t t i l di l d• Virtualization of infrastructure including cloud
• Mobile
13 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 14: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/14.jpg)
Agenda
• Why your customers won’t wait?
Wh t i li ti f t ti• What is realistic performance testing• Recommendations
14 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 15: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/15.jpg)
“Realistic testing” means testing from your customer’s point of viewcustomer s point of view
• Where they do it– LA, Edmonton, London, Bangkok, Des Moines,
Miami, Cannes, Dubrovnik, Osaka, Pusan
Wh h d• What they do– Key tasks, transactions, and business processes
• How they do it– Browsers galore, mobile devices
• When they do it (and how frequently)– Peak times, spikes
15 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
p
![Page 16: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/16.jpg)
The user’s distance from the content ttmatters
16 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 17: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/17.jpg)
Browsers render differently
17 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
Source: Actual Experience XF Real-User Monitoring Data; Gomez, Inc.
![Page 18: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/18.jpg)
What features should a realistic web performance testing platform have? It should:performance testing platform have? It should:
• Test from the end user’s point of view– Different locations
– Different browsers
– Different loads
– Different tasks
• Be easy for developers, testers and business people to use.
• Identify and resolve problems along the entire web application delivery chain.
18 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 19: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/19.jpg)
Agenda
• What do customers want?
Wh t i li ti f t ti• What is realistic performance testing
• Recommendations
19 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 20: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/20.jpg)
Recommendations (1 of 2)• Design a test plan based on how real users interact with your web application
• Use testing tools that allow you to test from your user’s point of view:
Wh Wh t H Wh– Where, What, How, When
• Test the entire Web application
• Democratize testing: – Build a test team that includes anyone who has a
stake in the outcome
20 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 21: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/21.jpg)
Recommendations (2 of 2)• Testing early to nip problems in the bud
• Test often to maintain a consistently supreme user experience
• Follow Blazing Fast Web Performance Best Practices to keep your Web site humming
21 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 22: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/22.jpg)
In this fiercely competitive environment, realistic web performance testing is the
right answer.
![Page 23: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/23.jpg)
Thank you
Mike Gualtieri
1 617 613 6145+1 617.613.6145
www.forrester.com
23 Entire contents © 2008 Forrester Research, Inc. All rights reserved.
![Page 24: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/24.jpg)
Gomez Profile
▪ First SaaS Web Application Experience Management Suite▪ Headquarters pe e ce a age e t Su te
▪ Web Performance and Load Testing
▪ Cross-Browser Testing
▪ Lexington, MA
▪ Global Offices Cross Browser Testing
▪ Web Performance Monitoring
▪ 80 000+ internet experience
▪ UK
▪ Germany ▪ 80,000+ internet experience measurement points in 162 countries
Ge a y
▪ Switzerland
▪ China▪ Over 2,500 Customers Worldwide
▪ 12 out of 20 most visited US Websites tili G
24
utilize Gomez
▪ 300 Employees
![Page 25: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/25.jpg)
Today’s Testing Challenges
Never Enough Time Small QA Teams
25Testing the End-User PerspectiveCatching All the Problems
![Page 26: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/26.jpg)
When To Load Test
Launching New Apps/Features Deploying New Infrastructure
26Penetrating New MarketsMajor Marketing Campaigns
![Page 27: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/27.jpg)
Web Application Delivery In A Web 2.0 World
▪ Web applications ppincreasingly assembled at browser level▪ 3rd party
performance from ads, videos, shopping cart vary widely cart vary widely under load
▪ Content Delivery Network configuration and effectiveness impact
d i
27
end-user experience
![Page 28: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/28.jpg)
Load 2.0 Testing Solutions
Overcome Limitations of 1.0 Testing Approaches
▪ Find and fix problems across the entire web application delivery chain▪ Inside and outside the firewall▪ Inside and outside the firewall
▪ Flexible and easy to useFlexible and easy to use▪ On-demand – no installation required▪ Highly scalable to accommodate
peak loadspeak loads▪ Supports Agile development
28
▪ Test from the end-user perspective
![Page 29: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/29.jpg)
Ensure Web Experiences Scale Under Load With Reality Load XF
Deploy web applications and ▪ Deploy web applications and infrastructure with confidence
▪ Combine high volume web application load testing with broad geographic experience testing
▪ Ensure on-demand testing gflexibility
▪ Detect failures in user experience not caught by traditional load not caught by traditional load testing
▪ Testing across the entire web li ti d li h i
29
application delivery chain
![Page 30: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/30.jpg)
Critical To Test Across Entire Web Application Delivery Chain
Key Load Testing Metrics• Success rate of end-user workflow• Response time of end-user workflowResponse time of end user workflow• Quality of end-user workflow delivery
30
![Page 31: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/31.jpg)
Load Testing 2.0Performance Testing For A Web 2.0 World
▪ Self-service testing gives you control to test as neededcontrol to test as needed
▪ Test internal & external web application components to ensure transactions perform ensure transactions perform under load
▪ Test cloud-based applications & servicesservices
▪ SaaS based Load Testing 2.0 solutions enable on-demand, scalable testing without
C bi hi h l g
associated hardware investment & maintenance costs
▪ Test the full web application
Combine high volume application load testing with broad geographic
31
ppdelivery chain all the way to your end users’ desktops
experience testing
![Page 32: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/32.jpg)
Critical To Test Across Entire Web Application Delivery Chain
22 33
11
![Page 33: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/33.jpg)
Ensuring Performance Of Login Process
Company• Online presence for a popular television show
11
p p p
Situation• Following episodes of the television show the web site
sees high traffic spikessees high traffic spikes
• Goal was to achieve 1500 logins per minute
• Load tested database optimizations aimed to improve performance in anticipation of another large traffic spikeperformance in anticipation of another large traffic spike
33
![Page 34: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/34.jpg)
Application Bottleneck Causes Immediate Response Time Issue
• As users were added the• As users were added, the response time of step 3 (the login) climbed immediately
• The test bottlenecked at 160 logins per minute (Goal 1500)
• But quickly dropped off as users received server errors
• New login query was not optimized and was bottlenecking the database
’ CPU
34
servers’ CPUs
![Page 35: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/35.jpg)
Ensuring Performance Of E-Commerce Transaction
22
Company• Online Retailer
Situation• Typical e-Commerce type application
S l 3 d P ti i l d i• Several 3rd Parties now involved in serving up key content
• Goal was to validate performance of entire application
![Page 36: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/36.jpg)
Response Times Rise Due To Third-Party Object Error
The load increases throughout the test
The transaction rate increases and then falls off as response times climb
Errors are seen, all on a 3rd party object
3 d t h d i ffi i t f ll d d li ti• 3rd party hardware was insufficient for overall demands on application• Based on SLAs 3rd party had to improve performance to get paid
![Page 37: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/37.jpg)
Ensuring Performance In Target Markets
33
Company• Regional Online News Source
Situation• Began testing for the 2008 election season• Goal was to validate performance of overall
application, and specifically for users in 2 key geographies
• Testing started with the homepage
![Page 38: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/38.jpg)
No Performance Issues Detected From Data-Center
There was only Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
Page response
1 page error and 11 errors total out of 60000+ transactions
Rate of 99%
Page response times stayed under 4 seconds, outside of one brief blip
transactions
p
By traditional test standards the test passed
![Page 39: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/39.jpg)
Performance Issues Detected From Real User Desktops
Key geographies for this customer are New York and Pennsylvania
Last Mile data showing substantial number of measurements greater than 4 seconds
![Page 40: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/40.jpg)
Performance Issues Detected For Target Market End Users
• Last Mile Users exceeding the 4-second goal – 29%
• Overall availability was 96.3%
• Availability for Last Mile UsersAvailability for Last Mile Users exceeding 4 seconds – 87%
• Last Mile Users reporting 10+ gsecond response times – 8%
B th t d d th t t f il dBy these standards the test failed
• Cutting 1 5 to 2 sec off the response time for the page would move• Cutting 1.5 to 2 sec. off the response time for the page would move another 16% of the Last Mile users into the 4 sec. response time range
![Page 41: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/41.jpg)
Gomez Offers Full Lifecycle SolutionsFor Web Experience Management
“Gomez has made spectacular progress in the last 18 months…to
41
p p gbecome the leader in web experience management.”
Jean-Pierre Garbani, Vice President, Principal Analyst, Forrester ResearchTech Horizons: Evaluating Gomez’s Web Experience Management Services, August 2008
![Page 42: Achieving Peak User Experiences & Optimizing Web Performance - Load Testing, Stress Testing & Performance Testing in an Agile Web 2.0 World](https://reader034.vdocuments.site/reader034/viewer/2022042700/554c9c06b4c905c10d8b4f82/html5/thumbnails/42.jpg)
Find Out How Gomez Can Help You
Some of Our CustomersEnsuring great experiences in a Web 2.0 world requires a new approach to web application load testing Reality Load XFapplication load testing - Reality Load XF
Reality Load XF• Self-service or full turnkey solution with Gomez
P f i l S i ff iProfessional Services offerings • Tests outside-in, with drill down to web application
components• SaaS with no investment or maintenance costsSaaS with no investment or maintenance costs
and rapid payback• Full desktop browser testing across globally
distributed geographies
Learn More:Reality Load XF Solution Briefhttp://www.gomez.com/pdfs/SB_RealityLoad.pdf
42Speak To A Gomez Solutions ExpertToll Free: 877.372.6732