Download - Functional Hostnames and Why they are Bad
![Page 1: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/1.jpg)
Puppet Camp April 9th 2013
![Page 3: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/3.jpg)
About Dropbox
• Thousands of instances/servers
• Mostly Python Stack
• EC2 and Large Self Datacenters
• Over a billion file syncs per day
• Thousands of MySQL Shards
• 4 SREs and 1 DBA
![Page 4: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/4.jpg)
A story of a startup...
Hostapuppet.com
![Page 5: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/5.jpg)
Chapter One
• 1 or 2 teams
• Couple of hosts
• Webserver and a database
• Maybe one ops guy
![Page 6: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/6.jpg)
What Ops People Like
• Simplicity
• Repeatability
• Assurances that things remain consistent
• Puppet / Configuration management
![Page 7: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/7.jpg)
Config Management
node 'www1.example.com' { include common include apache include squid } node 'db1.example.com' { include common include mysql }
![Page 8: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/8.jpg)
My First Puppet Config
Node ‘mickey.hostapuppet.com’ { include common include webserver include sudoers
} Node ‘donald.hostapuppet.com’{
include common include db include sudoers
}
![Page 9: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/9.jpg)
Sudoers Module
… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …
![Page 10: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/10.jpg)
Sudoers File itself
Host_Alias DONALD=donald.hostapuppet.com
Host_Alias MICKEY=mickey.hostapuppet.com
db_guy DONALD=(all) NOPASSWD: ALL
ops_guy MICKEY=(all) NOPASSWD: ALL
![Page 11: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/11.jpg)
Chapter Two: A growing service
• A few teams
• 2 or 3 services
• multiple types of hosts
– Web
– API
– DB
![Page 12: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/12.jpg)
Hostnames
• sjc-web[1-N]
• sjc-db[1-N]
• sjc-api[1-N]
![Page 13: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/13.jpg)
Host Regex
$hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)
![Page 14: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/14.jpg)
Hosttypes $hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)
if $hosttype == ‘web’ { include sudoers include web
} If $hosttype == ‘db’ {
include sudoers include db
}
if $hosttype == ‘api’ { include sudoers include api
}
![Page 15: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/15.jpg)
Back to sudoers
… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …
![Page 16: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/16.jpg)
Sudoers File itself
Host_Alias WEB=sjc-web*
Host_Alias DB=sjc-db*
Host_Alias API=sjc-api*
database_guy DB=(all) NOPASSWD: ALL
ops_guy WEB=(all) NOPASSWD: ALL
api_team API=(all) NOPASSWD: ALL
![Page 17: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/17.jpg)
Hypergrowth
0
20
40
60
80
100
120
0 2 3 5
Users (millions)
Users (millions)
![Page 18: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/18.jpg)
Chapter 3: An Expanding Infrastructure
• Lots of new hires!
• A bunch more developers
• Some PMs
• Some Designers
![Page 19: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/19.jpg)
All Kinds Of Problems…
• Boxes of same hardware class running
different services
• Boxes serving more than one role
(remember sudoers?)
• Deploying or moving hosts quickly
![Page 20: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/20.jpg)
Renaming a host
• Update dns
• Update dhcpd.conf
• Push both
• Update puppet configs
• Update code
![Page 21: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/21.jpg)
OMG I JUST RENAME HOSTS!
![Page 22: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/22.jpg)
Sudoers File From Chapter Two…
Host_Alias WEB= sjc-web* Host_Alias API=sjc-api* Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL
![Page 23: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/23.jpg)
Sudoers File in Chapter 3
Host_Alias WEB= sjc-web* Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553 Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL
![Page 24: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/24.jpg)
Dropbox
• We did all that.
• We’re still paying the taxes for doing
that.
• But there is a light at the end of the
tunnel…
![Page 25: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/25.jpg)
ABSTRACT THE SERVICE
FROM THE HOST!
![Page 26: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/26.jpg)
So what does that mean?
• Make hosts role agnostic
• Do not require invasive changes
• Simple interfaces
![Page 27: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/27.jpg)
Making hosts role agnostic
• Positional
• Serial Numbers
• Anything that doesn't change
![Page 28: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/28.jpg)
The Dropbox Plan
• Positional names
• Custom Machine Database
• External Node Classifier
• Transitioning Puppet configs
• Naming service(s) for convenient names
![Page 29: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/29.jpg)
Service/Machine Management Database
• Universal Source of Truth
• Manage roles / attributes
• Generated configs
- Gmond, Nagios, etc
![Page 30: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/30.jpg)
What exactly is the ENC
• External Node Classifier
• Inject variables (and other) from external
process
• YAML Output
![Page 31: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/31.jpg)
Part 2: External Node Encoders
![Page 32: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/32.jpg)
Sudoers++
• Move from monolithic to modular
• Includes! (Weird caveats)
• Just use ALL for Host_Lists
![Page 33: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/33.jpg)
Sudoers at Dropbox
![Page 34: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/34.jpg)
Part 3: Helper Functions
![Page 35: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/35.jpg)
Sudoers with tags
![Page 36: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/36.jpg)
Sudoers with tags
![Page 37: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/37.jpg)
Provisioning
• Preload MDB, DNS, DHCPD, etc.
- Set it and forget it
• Have spares ready for any roles
• Assigning a role is one command
• No more renames!
![Page 38: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/38.jpg)
Dynamic Naming w/ PowerDNS
![Page 39: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/39.jpg)
Dynamic Naming w/ PowerDNS
![Page 40: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/40.jpg)
Zookeeper
• ZKNS included with the Vitess project
• ZK is in use at various different companies (YouTube, Twitter, AirBnB)
![Page 41: Functional Hostnames and Why they are Bad](https://reader033.vdocuments.site/reader033/viewer/2022052903/557a0632d8b42a1c0a8b4cbf/html5/thumbnails/41.jpg)
Q&A
λ FAQ #1: Are you hiring? - Yes! Come talk to us. :)