Download - Web Performance Culture and Tools at Etsy
Web Performance Culture and Tools at Etsy
Mike Brittain@mikebrittain
Director of Eng., InfrastructureEtsy
All of this started in the last year.
Make PerformanceMatter
$0.20 per item3.5% transaction fee
Focus on the marketplace
ecommerce
ecommercead supported
ecommercead supportedsubscription
ecommercead supportedsubscription“free”
$$$
“Prove it.”
“Our own years of testing have conclusively shown that
when speed of a feature or product improves, usage,
quite simply, goes up.”
http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html
100+ web, api, jobs40+ databasesMany more doing lots of other stuff...
$$$
Performance is fun
I want you to tell mehow you feel...
mill
isec
onds
95th percentile
Average
mill
isec
onds This is when we
made it go faster!95th percentile
Average
I am better than you.
Make PerformanceMatter
PerformanceCulture
Our perf team is two people...
Measure
MeasureBuild tools
MeasureBuild toolsTrain & Coordinate
MeasureBuild toolsTrain & CoordinateMotivate
MeasureBuild toolsTrain & CoordinateMotivateDedicated
Our perf team is two people...
But every engineer thinksabout performance
Everything is slow. Please fix.
page generationtime
2011 Goal800 ms on five pages
http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/
http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/
Performance degradationis a P2 bug
Performance degradationis a P2 bug
WARNING: This part is hard
2012 Goal800 ms for all pages
Denormalize dataOptimize queriesRewrite from scratchCaching (memcache, app code)
Engineering
Network switchesOperating systemsNewer hardwareAdditional hardware capacity
Operations
Identify concerns earlyKill lagging features
Product
PerformanceCulture
Tools
How do you make the graphs?
Perf-related data belongs in your server logs
LogFormat %l %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\" %{custom_field}n
apache_note(“custom_field”, $whatever);
LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined
$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;
register_shutdown_function()
apache_note('php_microsec', $timer_diff);
web0060 66.249.71.110 - - [24/Aug/2011:04:16:52 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)" - 13399576 505780 554876
Using something else?time, http method, request uri, response code, referer, user-agent, response time, response memory, custom segmentation fields...
Quick averagesgrep "GET /listing/" access.log | \
awk '{sum=sum+$(NF-1)} END {print sum/NR}'
Quick graphsgrep "GET /listing/" access.log | \
perl -pe "s/.*\[.*\d{4}:(\d{2}):(\d{2}):\d{2}.*\]/\1:\2/" | \
awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat
gives you...
Quick graphs# /tmp/pagetimes.dat
18:37 251.018:38 252.118:39 253.518:40 251.018:45 250.0 and then...
Quick graphs# GNUPLOT
set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points
Quick graphs
LogsterRun by cronMaintains a cursor on log filesSimple parsing & aggregationOutput to Ganglia or Graphite
github.com/etsy
Logster & Graphite
Signed-in vs. Signed-out
Response timesby bucket
time (ms)
resp
onse
s (pa
ges)
“Look at your Data”John Rauser, Velocity 2011
http://www.youtube.com/watch?v=coNDCIMH8bk
Idea lifted fromhttp://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html
Don’t forget the API
Measure performancedata in Production
Profiling
Lightweight, inline profiling
Start and end times wrapped around service calls databases, memcache, apc, 3rd party services, etc.
Enable with query string
“I/O Profiler”
“I/O Profiler”
“I/O Profiler”
XHProfgithub.com/facebook/xhprof
Deployed on pre-prod machineDirect access to profilingEnable with query string
Profile againstProduction
Make performance matter
Start small
Make graphs and reports
Create simple tools
Celebrate performance wins
Jobs at EtsyWe are currently hiring for a number of positions in engineering and operations.
http://www.etsy.com/careers
