metrics driven engineering (velocity 2011)

Download Metrics driven engineering (velocity 2011)

Post on 12-Sep-2014

2.154 views

Category:

Technology

3 download

Embed Size (px)

DESCRIPTION

 

TRANSCRIPT

  • METRICS-DRIVENENGINEERING at

    Kellan Elliott-McCrea, VP of Eng.kellan@etsy.com @kellan

    Tuesday, June 5, 12

  • Tuesday, June 5, 12

  • Tuesday, June 5, 12

  • What is Etsy?

    Tuesday, June 5, 12

  • 8.5+ million items in the marketplace

    Tuesday, June 5, 12

  • 400,000+ active

    Tuesday, June 5, 12

  • $300+ million in sales in 2010

    ~$41 million/month

    Tuesday, June 5, 12

  • > $1000 / minute

    Tuesday, June 5, 12

  • > 1 billion page views / month

    Tuesday, June 5, 12

  • business in over 150 countries

    Tuesday, June 5, 12

  • deploy the site, every ~20 minutes

    Tuesday, June 5, 12

  • engineering team grew

    ~4x in 2010

    Tuesday, June 5, 12

  • Metrics?

    Tuesday, June 5, 12

  • Logs, Graphs, Trends,

    and Correlations

    Tuesday, June 5, 12

  • Metrics Driven?

    Tuesday, June 5, 12

  • Making Decisions

    Tuesday, June 5, 12

  • How many visitors are

    using this thing?

    Tuesday, June 5, 12

  • Can we deploy that to

    100% of our visitors?

    Tuesday, June 5, 12

  • Did we make it faster?

    Tuesday, June 5, 12

  • Did I just break something?

    Tuesday, June 5, 12

  • WHO MAKES THESE GRAPHS?

    Well, the Ops team manages the network, racks the servers, installed the

    monitoring tools, wears the pagers, blah, blah, blah...

    Q.A.

    Tuesday, June 5, 12

  • but... Engineers build

    the application.

    Tuesday, June 5, 12

  • Dev + Ops

    Tuesday, June 5, 12

  • ACCESS

    Tuesday, June 5, 12

  • Yes! No.

    Tuesday, June 5, 12

  • Engineers are too busy!

    Tuesday, June 5, 12

  • Heres the BIG SECRET...

    Tuesday, June 5, 12

  • ... MAKE IT EASY!

    Tuesday, June 5, 12

  • Simple, open source tools

    Tuesday, June 5, 12

  • Cacti (network, SNMP)Ganglia (machines)Graphite (application)Splunk (log analysis, nightly reports)Nagios (alerting)

    Tuesday, June 5, 12

  • Gancluster orientedhuge community contributed recipes2.0 released today (including several Flickr and Etsy patches!)gmetad makes it easy to track custom metrics

    Tuesday, June 5, 12

  • Tuesday, June 5, 12

  • Graphitesuper flexible collection and displayper metrics bucketssingle instance super easy to write and use custom display functions

    Tuesday, June 5, 12

  • Logging

    Tuesday, June 5, 12

  • Logger::log_error("User login failed. Reason: $msg for

    $username", login);

    Tuesday, June 5, 12

  • web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [14531658] User login failed. Reason: wrong

    password for ...

    Tuesday, June 5, 12

  • web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [14531658] User login failed. Reason: wrong

    password for ...

    Tuesday, June 5, 12

  • web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [14531658] User login failed. Reason: wrong

    password for ...

    Tuesday, June 5, 12

  • web0054 [Fri Mar 04 16:27:48 2011] [info] [login] [14531658] User login failed. Reason: wrong

    password for ...

    Tuesday, June 5, 12

  • web0054 [Fri Mar 04 16:27:48 2011] [info] [login] [14531658] User login failed. Reason: wrong

    password for ...

    Tuesday, June 5, 12

  • web0054 [Fri Mar 04 16:27:48 2011] [info] [login] [14531658] User login failed. Reason: wrong

    password for ...

    Tuesday, June 5, 12

  • Counting and Timinghttp://code.flickr.com/blog/2008/10/27/counting-timing/

    Tuesday, June 5, 12

  • Logster

    Tuesday, June 5, 12

  • Logsterhttps://github.com/etsy/logster

    Tuesday, June 5, 12

  • Forked from ganglia-logtailer :

    - Daemon mode (only cron mode) + Support for Graphite + Simplified parsing scripts

    Tuesday, June 5, 12

  • web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppppp!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling

    Tuesday, June 5, 12

  • Fatals Errors Warnings

    Tuesday, June 5, 12

  • runs out of cronmaintains a cursor into log filessupports ganglia and graphite custom parsers much easier to write then gmetad

    Tuesday, June 5, 12

  • Apache access logs

    Tuesday, June 5, 12

  • LogFormat "%h %l %u %t \"%r\" %>s %b" common

    Tuesday, June 5, 12

  • LogFormat "%{X-Forwarded-For}i %{True-Client-IP}i %l %u %t \"%r\" %>s %b

    \"%{Referer}i\" \"%{User-Agent}i\" %{etsy_shop_id}n %{etsy_uaid}n %V %

    {etsy_ab_selections}n %{etsy_request_uuid}n %

    {etsy_api_consumer_key}n %{etsy_api_method_name}n %

    {php_memory_usage_bytes}n %{php_time_microsec}n %D" combined

    Tuesday, June 5, 12

  • %{etsy_ab_selections}n

    Tuesday, June 5, 12

  • %{etsy_uaid}n

    Tuesday, June 5, 12

  • Graphs

    Tuesday, June 5, 12

  • If Engineering at Etsy has a religion, its the Church of Graphs. If it moves, we

    track it. - Erik Kastner

    http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

    Tuesday, June 5, 12

  • Tuesday, June 5, 12

  • StatsD

    Tuesday, June 5, 12

  • StatsDhttps://github.com/etsy/statsd/

    Tuesday, June 5, 12

  • StatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);

    Tuesday, June 5, 12

  • StatsD::timing("gearman.time", $msec);

    90th pct

    average

    lower

    Tuesday, June 5, 12

  • Ad hocname value timestamp

    Tuesday, June 5, 12

  • echo "events.deploy.site 1 `date +%s`" \| nc graphite.etsycorp.com 2003

    Tuesday, June 5, 12

  • Correlations

    Tuesday, June 5, 12

  • echo "events.deploy.site 1 `date +%s`" \| nc graphite.etsycorp.com 2003

    Tuesday, June 5, 12

  • Trends + Eventstarget=drawAsInfinite(events.deploy.site)

    Tuesday, June 5, 12

  • What Happened?

    Tuesday, June 5, 12

  • Holt-Winters

    Tuesday, June 5, 12

  • "Forecasting Sales by Exponentially Weighted Moving Averages". Peter

    Tuesday, June 5, 12

  • "Aberrant Behavior Detection in Time Series for Network Monitoring".

    Tuesday, June 5, 12

  • "Holt-Winters Forecasting Applied to Poisson

    Processes in Real-Time".

    Tuesday, June 5, 12

  • holtWintersConfidence(Upper|Lower)

    Tuesday, June 5, 12

  • holtWintersAberration

    Tuesday, June 5, 12

  • business metrics with confidence bands

    ==alertable business metrics

    Tuesday, June 5, 12

  • 16,000 metrics in GRAPHITE

    (plus 32,000 metrics in GANGLIA)

    Tuesday, June 5, 12

  • 16,000 metrics in GRAPHITE

    (plus 32,000 metrics in GANGLIA)

    Tuesday, June 5, 12

  • Dashboards

    Tuesday, June 5, 12

  • Dashboards

    Tuesday, June 5, 12

  • Dashboards

    Tuesday, June 5, 12

  • Hard

    Tuesday, June 5, 12

  • $g = new Graphite($time);$g->setTitle('File Not Found');$g->addMetric('webs.errorLog.notExist', '#00cc00');$g->showDeploys(true);echo $g->getDashboardHTML(280, 220);

    Easy!

    Tuesday, June 5, 12

  • 48 dashboards by32 engineers

    Tuesday, June 5, 12

  • Application health

    Tuesday, June 5, 12

  • High-level visibility

    Tuesday, June 5,