setting up new relic's infrastructure
DESCRIPTION
TRANSCRIPT
Setting up New Relic's Infrastructure: A bunch of things we did wrong and a few we did right.DAVE PETERSON & NIC BENDERS2013-10-25
David Peterson([email protected])
Nic Benders([email protected])
[“star” graph of code base at launch]
100,000req/sec
2Gbps
200TB
Who Does What?
Site Operations != IT
You are not Batman
Do your work during the day
Lowest hour is still 60% of peak hour!
Put your serverson the moon
Never open the box
Domain names are cheap
10.0.0.0/31 10.0.0.0-10.0.0.1 255.255.255.254 2
10.0.0.0/30 10.0.0.0-10.0.0.3 255.255.255.252 4
10.0.0.0/29 10.0.0.0-10.0.0.7 255.255.255.248 8
10.0.0.0/28 10.0.0.0-10.0.0.15 255.255.255.240 16
10.0.0.0/27 10.0.0.0-10.0.0.31 255.255.255.224 32
10.0.0.0/26 10.0.0.0-10.0.0.63 255.255.255.192 64
10.0.0.0/25 10.0.0.0-10.0.0.127 255.255.255.128 128
10.0.0.0/24 10.0.0.0-10.0.0.255 255.255.255.0 256
10.0.0.0/23 10.0.0.0-10.0.1.255 255.255.254.0 512
10.0.0.0/22 10.0.0.0-10.0.3.255 255.255.252.0 1024
10.0.0.0/21 10.0.0.0-10.0.7.255 255.255.248.0 2048
10.0.0.0/20 10.0.0.0-10.0.15.255 255.255.240.0 4096
10.0.0.0/19 10.0.0.0-10.0.31.255 255.255.224.0 8192
10.0.0.0/18 10.0.0.0-10.0.63.255 255.255.192.0 16384
10.0.0.0/17 10.0.0.0-10.0.127.255 255.255.128.0 32768
10.0.0.0/16 10.0.0.0-10.0.255.255 255.255.0.0 65536
Private IPsare FREE
Hardware
System Management
X6550 E5-2450 E5-2643 E5-2690
Average response time
137
ms
167
ms
158
ms
189
ms
25% performance improvement
2x 6-Core X6550@ 2.67 GHz
2x 8-Core E5-2690@ 2.90 GHz
Storage
√
Databases
32-bits isn't enoughclass CreateTribbles < ActiveRecord::Migration def change create_table :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end endend
32-bits isn't enough`id` int(11) NOT NULL AUTO_INCREMENT
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT
`id` SERIAL
Your database is not a queue
Long running querieswill kill you
And now, a thanks to the people who are out there every day fixing the mistakes that we made:
Jonathan Owens
Jonathan Thurman
Aaron Bento
Jonathan OwensJonathan Owens
Sean Kane
Paul Showalter
Karl Matthias