edge 2016 can webpagetest be trusted

© AKAMAI - EDGE 2016

Can Webpagetest be Trusted?Pierre Lermant

©2016 AKAMAI | FASTER FORWARDTM

WPT

It is THE synthetic performance testing tool

● Free to use public instanceThrough UI or APIs

● Feature rich

● Maintained to latest standards

● Can be hosted privately


Goal:Explore WPT public instances repeatability and consistency


● Thirty (30) URLs tested: Alexa top 10 from Shopping, Business, News

● Primarily focused on two metrics: onload time and speedIndex

● Chrome browser, 9 runs, medians, first view only, video on

● Tests ran between Sunday October 9 and Sunday October 16o All using APIs from python scriptso All URLs run in parallel from different agent locations

Experimental Setup


Reliability


Impact of location and retries on successful URLs (9 runs, 30 URLs)

Reliability


Lesson # 1:● Check number of successful runs

Especially from non EC2 locationsIf 9 runs needed, plan for a few retries


How many runs?


Impact of run number on main metrics, 30 URLs - Oct 14 run

How many runs?


Impact of run numbers on Coefficient of Variation (%) - Oct 14 run

How many runs?


Lesson # 2:● First run yields slightly higher

onload and speedIndex values● Variability seems to settle

around 10-15%, except for TTFB


Intra-Location consistency


Impact of concurrent runs in same locations on median onload time

Intra location consistency

Oct 13 Night Oct 14 Day Oct 16 Night


Impact of concurrent runs in same locations on median speedIndexIntra location consistency

Oct 13 Night Oct 14 Day Oct 16 Night


Lesson # 3:● Intra-location results within ~10 %


Inter-Location consistency


Impact of location (desktop fios) on median onload times

Inter location consistency

Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night


Impact of location (desktop fios) on median speedIndex times

Inter location consistency

Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night


Lesson # 4:● Results vary greatly with location

EC2 instances slower than private ?


RelativeInter-location consistency


Relative site speed ranking per location (%)

Relative consistency

● Dulles:Fios v.s. ec2-us-east-1:FiosOnload: 14 %speedIndex: 24 %

● ec2-us-west-1:Fios v.s. SanJose:FiosOnload: 15 %speedIndex: 15 %


Lesson # 5:● Relative speed ranking

(roughly) preserved


Accuracy


WPT results vs actual browser experience: onload times

Accuracy

Average Median Oct 16th run


Lesson # 6:● Location Dulles:Cable

(=default) closest to Home!


Emulation v.s. Real Device


Emulation vs real device (Moto. G) for various agents: onload timesEmulation v.s. Real Device

Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night

Emulation

Emulation


Emulation vs real for various locations: speedIndex timesEmulation v.s. Real Device

Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night

Emulation

Emulation


Lesson # 7:● Large (> 20%) discrepancies

between real device and emulation (Dulles + Motorola G)


More ...


Compare WPT onload with Rum onload

Compare behavior of public instances with private ones

Find out what’s causing discrepancies between locations

Throttle emulation CPUs to better approximate real devices

More ...


Summary

● Location has a huge impact on main performance metrics

● For a given location, allow for about 10-15% “repeatability margin”

● More testing is needed (surprise!)

edge 2016 can webpagetest be trusted

Technology