edge 2016 can webpagetest be trusted

© AKAMAI - EDGE 2016

Can Webpagetest be Trusted?Pierre Lermant

©2016 AKAMAI | FASTER FORWARDTM

WPT

It is THE synthetic performance testing tool

● Free to use public instanceThrough UI or APIs

● Feature rich

● Maintained to latest standards

● Can be hosted privately

©2016 AKAMAI | FASTER FORWARDTM

Goal:Explore WPT public instances repeatability and consistency

©2016 AKAMAI | FASTER FORWARDTM

● Thirty (30) URLs tested: Alexa top 10 from Shopping, Business, News

● Primarily focused on two metrics: onload time and speedIndex

● Chrome browser, 9 runs, medians, first view only, video on

● Tests ran between Sunday October 9 and Sunday October 16o All using APIs from python scriptso All URLs run in parallel from different agent locations

Experimental Setup

©2016 AKAMAI | FASTER FORWARDTM

Reliability

©2016 AKAMAI | FASTER FORWARDTM

Impact of location and retries on successful URLs (9 runs, 30 URLs)

Reliability

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 1:● Check number of successful runs

Especially from non EC2 locationsIf 9 runs needed, plan for a few retries

©2016 AKAMAI | FASTER FORWARDTM

How many runs?

©2016 AKAMAI | FASTER FORWARDTM

Impact of run number on main metrics, 30 URLs - Oct 14 run

How many runs?

©2016 AKAMAI | FASTER FORWARDTM

Impact of run numbers on Coefficient of Variation (%) - Oct 14 run

How many runs?

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 2:● First run yields slightly higher

onload and speedIndex values● Variability seems to settle

around 10-15%, except for TTFB

©2016 AKAMAI | FASTER FORWARDTM

Intra-Location consistency

©2016 AKAMAI | FASTER FORWARDTM

Impact of concurrent runs in same locations on median onload time

Intra location consistency

Oct 13 Night Oct 14 Day Oct 16 Night

©2016 AKAMAI | FASTER FORWARDTM

Impact of concurrent runs in same locations on median speedIndexIntra location consistency

Oct 13 Night Oct 14 Day Oct 16 Night

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 3:● Intra-location results within ~10 %

©2016 AKAMAI | FASTER FORWARDTM

Inter-Location consistency

©2016 AKAMAI | FASTER FORWARDTM

Impact of location (desktop fios) on median onload times

Inter location consistency

Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night

©2016 AKAMAI | FASTER FORWARDTM

Impact of location (desktop fios) on median speedIndex times

Inter location consistency

Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 4:● Results vary greatly with location

EC2 instances slower than private ?

©2016 AKAMAI | FASTER FORWARDTM

RelativeInter-location consistency

©2016 AKAMAI | FASTER FORWARDTM

Relative site speed ranking per location (%)

Relative consistency

● Dulles:Fios v.s. ec2-us-east-1:FiosOnload: 14 %speedIndex: 24 %

● ec2-us-west-1:Fios v.s. SanJose:FiosOnload: 15 %speedIndex: 15 %

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 5:● Relative speed ranking

(roughly) preserved

©2016 AKAMAI | FASTER FORWARDTM

Accuracy

©2016 AKAMAI | FASTER FORWARDTM

WPT results vs actual browser experience: onload times

Accuracy

Average Median Oct 16th run

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 6:● Location Dulles:Cable

(=default) closest to Home!

©2016 AKAMAI | FASTER FORWARDTM

Emulation v.s. Real Device

©2016 AKAMAI | FASTER FORWARDTM

Emulation vs real device (Moto. G) for various agents: onload timesEmulation v.s. Real Device

Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night

Emulation

Emulation

©2016 AKAMAI | FASTER FORWARDTM

Emulation vs real for various locations: speedIndex timesEmulation v.s. Real Device

Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night

Emulation

Emulation

©2016 AKAMAI | FASTER FORWARDTM

Lesson # 7:● Large (> 20%) discrepancies

between real device and emulation (Dulles + Motorola G)

©2016 AKAMAI | FASTER FORWARDTM

More ...

©2016 AKAMAI | FASTER FORWARDTM

Compare WPT onload with Rum onload

Compare behavior of public instances with private ones

Find out what’s causing discrepancies between locations

Throttle emulation CPUs to better approximate real devices

More ...

©2016 AKAMAI | FASTER FORWARDTM

Summary

● Location has a huge impact on main performance metrics

● For a given location, allow for about 10-15% “repeatability margin”

● More testing is needed (surprise!)