consumable (tasty, even) prepared for: using salt to … · using salt to make infrastructure...
TRANSCRIPT
![Page 1: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/1.jpg)
MM/DD/YY
YOUR TITLE HERE
PREPARED FOR:
PLACE LOGO HERE
2016/04/07
MONITORING EVOLVED
Using Salt to Make Infrastructure Consumable (Tasty, Even)
Warren Turkal, Infrastructure h4X0r
![Page 2: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/2.jpg)
Introduction
• What SignalFx Does• How We Organize Our Infrastructure• Tooling• Why We Chose Saltstack• Integrating Salt with Our Infra• Practical Choices• Monitoring• Q&A
![Page 3: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/3.jpg)
What SignalFx Does
Monitoring Modern Infrastructure (as a Service)
• Consume metrics at any resolution (down to 1s) • At most any scale (many many 10s of billions of datapoints a day)• That are multi-dimensional (service, region, device type, canary, ...)• For: visualization, ad hoc exploration, statistical analysis, alerting
• Special sauce: in-stream, “real-time” analytics• To build composite signals out of raw metrics → meaningful alerts • On current data as it’s arriving and historical data• example: alert on (defined) week-over-week deviation in service-
level metric like 90th percentile successful response latency over moving 1-hour window
![Page 4: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/4.jpg)
How Our Infrastructure is Organized
Region (aws us-east-1)
![Page 5: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/5.jpg)
How Our Infrastructure is Organized
Region (aws us-east-1)
Culture (aa) Culture (bb)
![Page 6: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/6.jpg)
How Our Infrastructure is Organized
Region (aws us-east-1)
Culture (aa) Culture (bb)
Cell (aaab)
Cell (aaaa)
Cell (bbab)
Cell (bbaa)
![Page 7: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/7.jpg)
How Our Infrastructure is Organized
Region (aws us-east-1)
Culture (aa) Culture (bb)
Cell (aaab)
Host
Cell (aaaa)
Host
Cell (bbab)
Cell (bbaa)
HostHost
![Page 8: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/8.jpg)
How Our Infrastructure is Organized
Design goals:• Cells fail independently• TCP/IP connectivity between every host• Ability to choose placement of hosts that take advantage of
cost/latency/reliability concerns
![Page 9: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/9.jpg)
How Our Infrastructure is Organized
I gave a talk at SRECon15 with many more details: link
![Page 10: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/10.jpg)
Tooling - sfinfra
A library and a set of tools:• sf-culture-tool• sf-cell-tool• sflb• sfhost
![Page 11: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/11.jpg)
Starting from Zero Production - Why We Chose Salt
• Python• Extensible
![Page 12: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/12.jpg)
Starting from Zero - Why We Chose Salt
• Extensible• Python• Ponies
![Page 13: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/13.jpg)
Integration With Our Infra
Custom grains and pillars
![Page 14: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/14.jpg)
Integration With Our Infra
Custom states• HAProxy• SSH
![Page 15: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/15.jpg)
Practical Concerns
• one active salt-master in each culture• ability to bootstrap a master from a different culture• disaster recovery• minimal use of public formulas
![Page 16: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/16.jpg)
Monitoring
• Collectd plugin• Important Metrics
• Current highstates longer than 5m• Num nodes down (somewhat inaccurate and occasionally
wildly fluctuates)
![Page 17: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/17.jpg)
Future Work for Me
• master-tops• move to saltstack packages• investigate the ec2 external pillar module• investigate multi-master again
![Page 18: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/18.jpg)
Features I Would Like to See
• stackable pillars• something like master tops for pillar data files• use a stack of git branches to make up an environment’s
fileserver contents• port AWS related functionality to boto3 (especially s3fs)
![Page 19: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/19.jpg)
Conclusion
• Our tooling ensures hosts are brought up in a normalized way• Salt has some extension possiblities• Salt has taken us from 0 to 600ish hosts in a each culture with
lots of automation• Salt is working pretty well for us.
![Page 20: Consumable (Tasty, Even) PREPARED FOR: Using Salt to … · Using Salt to Make Infrastructure Consumable (Tasty, Even) Warren Turkal, Infrastructure h4X0r. Introduction • What SignalFx](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b5b738f7f8b9a302a8e018c/html5/thumbnails/20.jpg)
MM/DD/YY
YOUR TITLE HERE
PREPARED FOR:
PLACE LOGO HERE
THANK YOU!Follow me on Twitter @w00tSpeaks
Come see us at the booth for funtastic new features.
SIGN UP FOR A TRIAL AT:
signalfx.com