Download - Load balancing in the SRE way
![Page 1: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/1.jpg)
Load Balancing in the SRE way
Ke Zhu @shawnzhu Site (Un)Reliability Engineer at IBM
![Page 2: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/2.jpg)
For What?• GitHub Enterprise Cluster
• On Internet
• Zero downtime
• 100M+ HTTP requests per week
• 30k+ attacks per week
• 26k+ git clone per hour(https://help.github.com/enterprise/2.8/admin/guides/installation/maintenance-mode/)
![Page 3: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/3.jpg)
![Page 4: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/4.jpg)
Design Goals• Scripting Platform
• traffic conducting via code
• do social coding
• Observable
• Blue/Green deployment
• High performance
• Security from day one
![Page 5: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/5.jpg)
Software Stack
&& 🎩 magic kernel parameters in /etc/sysctl.conf 🐰
![Page 6: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/6.jpg)
Scripting Platform• OpenResty (Nginx + Lua) - https://openresty.org/en/
(Example: customized request rate limiting)
![Page 7: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/7.jpg)
Blue/Green Deployment• Can not terminate any TCP connection
• Two stacks:
• load-balancer-green
• load-balancer-blue (for experiment)
• Cloud DNS
• Switching A record + short TTL (~5m)
• Simple/Weighted Routing policy
• Run experiment by using docker image tags
• Real time metrics collection by librato.com
![Page 8: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/8.jpg)
• Test docker images
• RSpec + Serverspec
• Travis CI
• Test docker host
• RSpec + Serverspec
• Test Kitchen
Test Driven for Container
![Page 9: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/9.jpg)
❤vault• Secret mgmt via API - https://www.vaultproject.io/
• retrieve all secrets for provisioning load balancer via a single token with TTL 5min
![Page 10: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/10.jpg)
Blocking mode in Production• Signal Sciences - https://signalsciences.net/
![Page 11: Load balancing in the SRE way](https://reader035.vdocuments.site/reader035/viewer/2022070603/5870d2e11a28ab64768b5e4b/html5/thumbnails/11.jpg)
Summary
• Conducting HTTPS traffic via Lua code
• Blue-green deployment of Load balancer via DNS
• Testing docker with RSpec + Serverspec
• SignalSciences