sam haskins -...
Post on 13-Aug-2020
7 Views
Preview:
TRANSCRIPT
Continuous Deployment
Sam Haskins
Velocity China 2012
Etsy.comshaskins@etsy.com
Friday, December 21, 12
Friday, December 21, 12
20,000,000 Members
Friday, December 21, 12
800,000 Active Shops
Friday, December 21, 12
17,000,000 items
Friday, December 21, 12
0
150
300
450
600
2005 2006 2007 2008 2009 2010 2011
525.6
306.7
176.8
87.3
263.80.17
Millions of $
Friday, December 21, 12
0
150
300
450
600
2005 2006 2007 2008 2009 2010 2011
525.6
306.7
176.8
87.3
263.80.17
Millions of $
Friday, December 21, 12
(deploys/day)
Friday, December 21, 12
0
150
300
450
600
2005 2006 2007 2008 2009 2010 2011
525.6
306.7
176.8
87.3
263.80.17
Millions of $
Friday, December 21, 12
200 deployers
Friday, December 21, 12
200 deployersEngineers?
Friday, December 21, 12
200 deployersEngineers +Designers
Friday, December 21, 12
200 deployersEngineers +Designers +Product
Friday, December 21, 12
33 deploys/day
Friday, December 21, 12
33 deploys/day(average that includes weekends)
Friday, December 21, 12
33 deploys/day(average that includes weekends)
(just the web stack)
Friday, December 21, 12
Continuous Deployment
Friday, December 21, 12
Take the ceremonyout of releases
Friday, December 21, 12
Smaller changes,more often
Friday, December 21, 12
Fewer changes/deploy=
Fewer things thatcan go wrong
Friday, December 21, 12
Fewer changes/deploy=
Easier to see effects of your change
Friday, December 21, 12
Friday, December 21, 12
How to implement
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
<?php// search engine
$results = Search::use_grep($_GET[‘query’]);print_r($results);
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
$ curl http://shaskins.dev.etsy.com/search.php?query=bowties
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
$ scp -r ~/code webserver.prod.etsy.com:/docroot
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
$ tail -f access_log[15/Nov/2012:17:14:41 +0000] "GET /search?query=bowties" 200 [15/Nov/2012:17:14:42 +0000] "GET /search?query=knit+hat" 200[15/Nov/2012:17:14:45 +0000] "GET /search?query=vintage+chair" 200
Friday, December 21, 12
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
“Always Ship Trunk”Paul Hammondhttp://www.paulhammond.org/2010/06/trunk/
Friday, December 21, 12
Centralized configuration file
Friday, December 21, 12
<?php// Site configuration
$config[‘search’] = array( ‘enabled’ => ‘on’ ));
Friday, December 21, 12
<?php// search engine
$results = Search::use_grep($_GET[‘query’]);print_r($results);
Friday, December 21, 12
<?php// search engineif (!Feature::isEnabled(‘search’)) { header(‘HTTP/1.1 503 Service Unavailable’);}
$results = Search::use_grep($_GET[‘query’]);print_r($results);
Friday, December 21, 12
<?php// Site configuration
$config[‘search_method’] = array( ‘grep’ => ‘100%’
));
Friday, December 21, 12
<?php// Site configuration
$config[‘search_method’] = array( ‘grep’ => ‘100%’, ‘solr_service’ => ‘0%’ ));
Friday, December 21, 12
<?php// search engineif (!Feature::isEnabled(‘search’)) { header(‘HTTP/1.1 503 Service Unavailable’); exit();}
$results = Search::use_grep($_GET[‘query’]);
print_r($results);Friday, December 21, 12
<?php// search engineif (!Feature::isEnabled(‘search’)) { header(‘HTTP/1.1 503 Service Unavailable’); exit();}
switch (Feature::variant(‘search_method’)) { case ‘grep’: $results = Search::use_grep($_GET[‘query’]); break;
case ‘solr_service’: $results = Search::use_solr($_GET[‘query’]); break;}
print_r($results);Friday, December 21, 12
Lots of types of options
Friday, December 21, 12
<?php// Site configuration
$config[‘new_feature’] = array( ‘enabled’ => ‘0%’, ‘admin’ => ‘on’ ));
Friday, December 21, 12
<?php// Site configuration
$config[‘new_feature’] = array( ‘enabled’ => ‘0%’, ‘users’ => array( ‘test_user’, ‘big_seller’, ‘best_friend’ ) ));
Friday, December 21, 12
<?php// Site configuration
$config[‘new_feature’] = array( ‘enabled’ => ‘0%’, ‘groups’ => array( 1234, 6742 ) ));
Friday, December 21, 12
<?php// Site configuration
$config[‘new_feature’] = array( ‘enabled’ => array( ‘variant_1’ => ’15%’, ‘variant_2’ => ’20%’, ‘control‘ => ’65%’ ) ));
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
Friday, December 21, 12
Development Staging Production
Friday, December 21, 12
Development
Friday, December 21, 12
DevelopmentSeparate database
Friday, December 21, 12
DevelopmentSeparate databaseSeparate network
Friday, December 21, 12
Staging and Production
Friday, December 21, 12
Staging and ProductionSame databases
Friday, December 21, 12
Staging and ProductionSame databasesSame caches
Friday, December 21, 12
Staging and ProductionSame databasesSame cachesSame services
Friday, December 21, 12
Staging and ProductionSame databasesSame cachesSame servicesSame hardware
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
_________ ______ _____ ___________ /_____ ________ ___ /______ _____ _____(_)_______ ______ ___ /_______ _________ __ / _ _ \___ __ \__ / _ __ \__ / / /__ / __ __ \_ __ `/_ __/_ __ \__ ___// /_/ / / __/__ /_/ /_ / / /_/ /_ /_/ / _ / _ / / // /_/ / / /_ / /_/ /_ /\__,_/ \___/ _ .___/ /_/ \____/ _\__, / /_/ /_/ /_/ \__,_/ \__/ \____/ /_/ /_/ /____/
Friday, December 21, 12
Friday, December 21, 12
Configuration-onlydeploys
Friday, December 21, 12
code
test
release
feedback
Friday, December 21, 12
Metrics
Friday, December 21, 12
MetricsHundreds of thousands of metrics
Friday, December 21, 12
Business Metrics
Friday, December 21, 12
Business MetricsNumber of sales
Friday, December 21, 12
Business MetricsNumber of salesDollars of revenue
Friday, December 21, 12
Business MetricsNumber of salesDollars of revenueSources of traffic
Friday, December 21, 12
Application Metrics
Friday, December 21, 12
Application MetricsApplication actions
Friday, December 21, 12
Application MetricsApplication actionsUsage patterns
Friday, December 21, 12
Application MetricsApplication actionsUsage patternsTimes events happen
Friday, December 21, 12
System Metrics
Friday, December 21, 12
System MetricsCPU usage
Friday, December 21, 12
System MetricsCPU usageNetwork bandwidth
Friday, December 21, 12
System MetricsCPU usageNetwork bandwidthDisk IO
Friday, December 21, 12
Make metricseasy to create
Friday, December 21, 12
<?php// search engine
$results = Search::use_grep($_GET[‘query’]);print_r($results);
Friday, December 21, 12
<?php// search engine
$results = Search::use_grep($_GET[‘query’]);print_r($results);StatsD::increment(‘search.queries’);
Friday, December 21, 12
Friday, December 21, 12
Put them into dashboards
Friday, December 21, 12
Logs
Friday, December 21, 12
Friday, December 21, 12
Make metrics from logs
Friday, December 21, 12
Big-data analytics
Friday, December 21, 12
Culture of trust
Friday, December 21, 12
Tools solve technological friction
Friday, December 21, 12
Culture solves organizational friction
Friday, December 21, 12
Get rid of barriers
Friday, December 21, 12
Get rid of barriers“Talk to ops to get access to the logs”
Friday, December 21, 12
Get rid of barriers“Talk to ops to get access to the logs”“Get approval from CTO to push code”
Friday, December 21, 12
Get rid of barriers“Talk to ops to get access to the logs”“Get approval from CTO to push code”“Release manager pushes the buttons”
Friday, December 21, 12
Take responsibility
Friday, December 21, 12
Something’s broken?
Friday, December 21, 12
Something’s broken?Ops problem?
Friday, December 21, 12
Something’s broken?Ops problem?Developer problem?
Friday, December 21, 12
Something’s broken?Ops problem?Developer problem?Product problem?
Friday, December 21, 12
Something’s broken?Ops problem?Developer problem?Product problem?
Friday, December 21, 12
Something’s broken?Ops problem?Developer problem?Product problem?Problem.
Friday, December 21, 12
Everyone is responsible
Friday, December 21, 12
Dev/Ops/ProductCollaboration
Friday, December 21, 12
Blameless Postmortems
Friday, December 21, 12
Why Etsy Loves Continuous Deployment
Friday, December 21, 12
Features getdeveloped faster
Friday, December 21, 12
Iterative Development
Friday, December 21, 12
Increased confidence
Friday, December 21, 12
Data-backed decisions
Friday, December 21, 12
Better communication
Friday, December 21, 12
Mean time to recovery
Friday, December 21, 12
Tools Mentioned:•Feature flag tool (not yet released, but soon!)•Jenkins (http://jenkins-ci.org)•Deployinator (https://github.com/etsy/deployinator)•StatsD (https://github.com/etsy/statsd)•Graphite (http://graphite.wikidot.com)•Ganglia (http://ganglia.sourceforge.net)•Logster (https://github.com/etsy/logster)•Supergrep (https://github.com/etsy/supergrep)•Much more at https://github.com/etsy
Friday, December 21, 12
Thanks!
Friday, December 21, 12
top related