edge 2014: a modern approach to performance monitoring
DESCRIPTION
A Modern Approach to Performance Monitoring by Cliff Crocker, VP of Product Management, SOASTA "How fast are you? How fast should you be? How do you get there? In this talk Cliff will discuss traditional approaches to performance measurement and introduce a ""RUM First"" methodology. This approach begins with capturing performance directly from the end user as the single source of truth for cross-functional organizations focused on performance. Along the way, you will discover the relationship between RUM and synthetic monitoring, learn what to measure and how to capture it and finally how perceived performance impacts human behavior and your bottom line. Akamai Edge is the premier event for Internet innovators, tech professionals and online business pioneers who together are forging a Faster Forward World. At Edge, the architects, experts and implementers of the most innovative global online businesses gather face-to-face for an invaluable three days of sharing, learning and together pushing the limits of the Faster Forward World. Learn more at: http://www.akamai.com/edgeTRANSCRIPT
Cliff Crocker @cliffcrocker SOASTA VP Product, mPulse
1. How fast am I? 2. How fast should I be?
3. How do I get there?
Synthe'c vs. RUM
Synthetic 101 Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations. • A set ‘path’ or URL is defined • Tests are run either adhoc or scheduled and data is collected
RUM 101 Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.
• Involves instrumentation of your website via JavaScript
• Measurements are fired across the network to a collection point through a small request object (beacon)
<JS> <beacon>
RUM Cast a wide net • Identify key areas of concern • Understand real user impact • Map performance to human behavior & $$
Synthetic Diagnostic tool • Identify issues in a ‘lab’/remove variables • Reproduce a problem found with RUM
h0p://www.flickr.com/photos/84338444@N00/with/3780079044/
h0p://www.flickr.com/photos/ezioman/
The Early Days of RUM
• Round-trip time • Start/stop timers via JavaScript • Early contributors:
• Steve Souders/Episodes • Philip Tellis/Boomerang.js • Both widely in use today
Navigation Timing
Browser Support for Navigation Timing
!
10 © 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation Timing
DNS: Domain Lookup Time
function getPerfStats() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart}; }
11 © 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation Timing
TCP: Connection Time to Server
function getPerfStats() { var timing = window.performance.timing; return { connect: timing.connectEnd - timing.connectStart}; }
12 © 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation Timing
TTFB: Time to First Byte
function getPerfStats() { var timing = window.performance.timing; return { ttfb: timing.responseStart - timing.connectEnd}; }
13 © 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation Timing
Base Page
function getPerfStats() { var timing = window.performance.timing; return { basePage: timing.responseEnd - timing.responseStart}; }
14 © 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation Timing
Front-end Time
function getPerfStats() { var timing = window.performance.timing; return { frontEnd: timing.loadEventStart - timing.responseEnd}; }
15 © 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation Timing
Page Load Time
function getPerfStats() { var timing = window.performance.timing; return { load: timing.loadEventStart - timing.navigationStart}; }
Measuring Assets
• Strength of synthetic • Full visibility into asset performance
• Images • JavaScript • CSS • HTML
• A lot of which is served by third-parties • CDN!
Object Level RUM
Browser Support for Resource Timing
CORS: Cross-Origin Resource Sharing
Timing-Allow-Origin = "Timing-Allow-Origin" ":" origin-list-or-null | "*"
Start/End time only unless Timing-Allow-Origin HTTP Request Header defined
Resource Timing
var rUrl = ‘http://www.akamai.com/images/img/cliff-crocker-dualtone-150x150.png’; var me = performance.getEntriesByName(url)[0]; var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart }
Resource Timing
• Slowest resources • Time to first image • Response time by domain • Time a group of assets • Response time by initiator type (element type) • Cache-hit ratio for resources
For examples see: http://www.slideshare.net/bluesmoon/beyond-page-level-metrics
Resource Timing
• PerfMap - https://github.com/zeman/perfmap • Mark Zeman
• Waterfall.js - https://github.com/andydavies/waterfall • Andy Davies
1. How fast am I? 2. How fast should I be?
3. How do I get there?
Picking a Number
• Industry benchmarks? • Apdex? • Analyst report? • Case studies?
“Synthetic monitoring shows you how you relate to your competitors, RUM shows you how you relate to your customers.”
– Buddy Brewer
Benchmarking
Benchmarking
• http://soasta.com/mpulseUX
Benchmarking
• Page construction • Requests • Images • Size
• Other important metrics • Speedindex • Start Render • PageSpeed/Yslow scoring
Performance is a Business Problem"
"
Yahoo! - 2008
Increase of 400ms causes 5-9% increase in user abandonment
http://www.slideshare.net/stubbornella/designing-fast-websites-presentation
Shopzilla - 2009
A reduction in Page Load time of 5s increased site conversion 7-12%!
http://assets.en.oreilly.com/1/event/29/Shopzilla%27s%20Site%20Redo%20-%20You%20Get%20What%20You%20Measure%20Presentation.ppt
Walmart - 2012
http://minus.com/msM8y8nyh#1e
SF WebPerf – 2012 Up to 2% conversion drop for every additional second
So What?"""
SIMULATION
How Fast Should You Be?
• Use synthetic measurement for benchmarking your competitors
• Understand how fast your site needs to be to reach business goals/objectives with RUM
• You must look at your own data
1. How fast am I? 2. How fast should I be?
3. How do I get there?
Real users are not normal
Page Load Times
2.76s – Median
10.45s – p95
17.26s – p98
Platform Median 95th Percentile
98th Percentile
Mobile 3.62s 12.53s 20.04s
Desktop 2.44s 9.31s 15.86s
Page Load Times
Page Load Times
OS Median 95th Percentile
98th Percentile
Windows 7 2.41s 9.29s 15.89s
Mac OS X/10 2.30s 8.11s 13.45s
iOS7 3.27s 10.64s 15.79s
Android 4 4.06s 14.30s 27.93s
iOS8 3.53s 11.54s 19.72s
Windows 8 2.67s 10.75s 18,74s
Other Factors
• Geography • User Agent • Connection Type • Carrier/ISP • Device Type
Not All Pages are Created Equal
For a typical eCommerce site, conversion rate
drops by up to 50% when “browse” pages increase
from 1 to 6 seconds
Not All Pages are Created Equal
However, there is much less impact
to conversion when “checkout” pages degrade
How Do I Get There?
• Focus on the highest value opportunities/demographics • Identify key pages that have the most impact on your
KPIs • Prioritize based on reducing friction within the funnel or
critical path
Thank You!"""
Images
1. Modern (Title Slide): https://www.flickr.com/photos/looking4poetry/3477854720/in/photolist-6ijV3q-9t3sjb-cj8VWE-MNjPA-4yBqug-2rk5he-4nZVzJ-4yE45J-9t3Fvj-2rptr5-69ymdr-6XVTH-6QZ4dg-9t3HN5-8geUkB-6oEXs5-eH9zS-H2XTt-cKdJvb-7GNKWx-cKdKZd-5Rw3qt-jfT5Dx-Jfiuy-bTop6R-2m8kAB-Jfiuu-7E8eMf-9jqD6-9hYAvd-Jfius-gsqr7U-7KiNAZ-8kzG9V-euXdb-44qJNN-47jBuY-nsxZrZ-7cGjQL-4cfHKq-cGzbSC-aQtWPR-8y3sR3-6okjAW-5A7BSC-6aRsvh-eSMHkX-kxvdyK-9t3udu-c8EkYu
2. Man vs. Machine: https://www.flickr.com/photos/eogez/3289851965/in/photolist-61HmnB-9nkQu3-dfinF1-4GjK24-4GjL6Z-4GoV6C-dibVCU-4GoTXo-4GjF3k-4GjH7Z-4GoR6h-ek6eT9-ek6eUU-4fADUx-4fEDV9-65JA6W-65Jsxu-65JDnj-4fAE12-65JDTh-65JBZu-65Ek9M-65JtDu-65JuiN-65JzFS-65EfiH-65Jv63-65JxX7-65Enyc-65Eh4c-65Jxjm-65JwmU-65EhUP-65JCML-65EjxP-65JBQf-65Ee4Z-65JD4b-65JBF1-65Ec1r-65EkPB-65Jy4o-65Eg96-65JAjb-65Emfe-65Efoz-65JBam-65Eniv-65JCrU-65Eed2
3. https://www.flickr.com/photos/bradybd/2818154005/
4. https://www.flickr.com/photos/perspective/149321089/
5. Dartboard: http://en.wikipedia.org/wiki/Darts#mediaviewer/File:Darts_in_a_dartboard.jpg
References:
1. http://calendar.perfplanet.com/2011/a-practical-guide-to-the-navigation-timing-api/
2. http://www.slideshare.net/bluesmoon/beyond-page-level-metrics