diy synthetic: private webpagetest magic

DIY Synthetic: Private WebPagetest MagicVelocity NY 2013Jonathan Klein@jonathanklein

Wednesday, October 16, 13

Slides, Linksjkle.in/wpt


About Me


About Me•Performance Engineer At Etsy


About Me•Performance Engineer At Etsy• I write the Etsy Site Performance Reports


About Me•Performance Engineer At Etsy• I write the Etsy Site Performance Reports•Organize Boston Web Perf Meetup Group


Etsy Stats


Etsy Stats•World’s largest handmade marketplace


Etsy Stats•World’s largest handmade marketplace• 1.5 billion page views/month


Etsy Stats•World’s largest handmade marketplace• 1.5 billion page views/month•Almost $1B in sales last year


+160k to a page = +12% bounce rate

on mobile


Synthetic Monitoring


Synthetic Monitoring Real User Monitoring

Browser Instrumentation Navigation Timing API

Consistent trending over time

Highly variable

Largely in your control Last mile difficulties

Great for identifying regressions

Great for comparing across geographies/browsers

Not super realistic “Real User Monitoring”


$$$$


Background


EC2 Test Agents


No Built-in Automation


wpt-script


Motivation


Why We Use It


Why We Use It•Wanted long term synthetic trending


Why We Use It•Wanted long term synthetic trending• Automated tests


Why We Use It•Wanted long term synthetic trending• Automated tests•More control


Why We Use It•Wanted long term synthetic trending• Automated tests•More control• Low cost


Easy Configuration


{ "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log"}


# Run Test*/25 * * * * php run.php


# Run Test*/25 * * * * php run.php -c foo.conf


# Run Test*/25 * * * * php run.php -c foo.conf

# Get Results* * * * * php get_results.php


Advanced Features


{ "server": "http://example.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.webpagetest.org" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log", "prepend": "Login", "username": "someuser", "password": "somepassword", "run_options": { "video": 0 }}


{ "server": "http://example.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.webpagetest.org" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log", "prepend": ["BlockThirdParty", "bypass_cdn.txt"], "run_options": { "video": 0 }}


foreach ($prepends as $prepend) { if (file_exists($scripts_dir . $prepend)) { $script .= file_get_contents($scripts_dir . $prepend); } elseif (method_exists($this, $prepend)) { $script .= $this->$prepend(); }}


setDns site.etsystatic.com 38.106.64.123setDns img0.etsystatic.com 38.106.64.125setDns img1.etsystatic.com 38.106.64.125setDns img2.etsystatic.com 38.106.64.125setDns img3.etsystatic.com 38.106.64.125setDns ny-image0.etsy.com 38.106.64.125setDns ny-image1.etsy.com 38.106.64.125setDns ny-image2.etsy.com 38.106.64.125setDns ny-image3.etsy.com 38.106.64.125

bypass_cdn.txt


Gathering Results


<?php

// Splunk$splunkLogger = new SplunkLogger($config['splunkLog'], $logging_ns);foreach ($results as $result) { $splunkLogger->log($result);}

// Graphite$grapher = new Grapher($config['graphite'], $logging_ns);$grapher->graphResults($results);


https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metricsWednesday, October 16, 13

https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics

https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics

Disadvantages vs. Paid Tools


Disadvantages vs. Paid Tools•Harder to set up/maintain


Disadvantages vs. Paid Tools•Harder to set up/maintain• Functionality


Disadvantages vs. Paid Tools•Harder to set up/maintain• Functionality•Regions


Disadvantages vs. Paid Tools•Harder to set up/maintain• Functionality•Regions• Full page oriented


Disadvantages vs. Paid Tools•Harder to set up/maintain• Functionality•Regions• Full page oriented• Less consistent


Advantages vs. Paid Tools


Advantages vs. Paid Tools•Cost


Advantages vs. Paid Tools•Cost• Flexibility


Advantages vs. Paid Tools•Cost• Flexibility•People know the tool


Advantages vs. Paid Tools•Cost• Flexibility•People know the tool•Multi-page flows


Advantages vs. Paid Tools•Cost• Flexibility•People know the tool•Multi-page flows• Scripting


Our Use Cases


WebPagetest Catchpoint


WebPagetest Catchpoint•Multi-page flows


WebPagetest Catchpoint•Multi-page flows• Scripted tests


WebPagetest Catchpoint•Multi-page flows• Scripted tests• High volume testing


WebPagetest Catchpoint•Multi-page flows• Scripted tests• High volume testing• Ad-hoc tests



• DNS



• DNS• API



• DNS• API• Single Object Tests



• DNS• API• Single Object Tests•Multiple locations



• DNS• API• Single Object Tests•Multiple locations• Comparisons


Future Work


Improving the Process


Improving the Process•Manage disk space


Improving the Process•Manage disk space•Multiple regions


Improving the Process•Manage disk space•Multiple regions•More canned scripts


Improving the Process•Manage disk space•Multiple regions•More canned scripts•Automate upgrades


Leveraging WebPagetest


Leveraging WebPagetest• Front-end SPOF


Leveraging WebPagetest• Front-end SPOF•User Timings (AFT)


Leveraging WebPagetest• Front-end SPOF•User Timings (AFT)• Speed Index


Improving WebPagetest


WebPagetest + Vagrant


WebPagetest + Vagrant• Install VirtualBox


WebPagetest + Vagrant• Install VirtualBox• Install Vagrant


WebPagetest + Vagrant• Install VirtualBox• Install Vagrant• https://gist.github.com/jklein/6992296


https://gist.github.com/jklein/6992296


WebPagetest + Vagrant• Install VirtualBox• Install Vagrant• https://gist.github.com/jklein/6992296•vagrant up




WebPagetest + Vagrant• Install VirtualBox• Install Vagrant• https://gist.github.com/jklein/6992296•vagrant up

• http://localhost:8080




http://localhost:8080

http://localhost:8080

800MB


webpagetest-api


Both Are Good Options


Results > Tools


Thanks!


@jonathanklein

[email protected]

www.etsy.com/careers

Get in Touch


http://www.etsy.com/careers

http://www.etsy.com/careers

diy synthetic: private webpagetest magic

Technology