just enough web ops for web developers
DESCRIPTION
Datadog is monitoring that does not suck. It's metrics friendly, people friendly and developer friendly monitoring. Learn more at https://www.datadoghq.com/TRANSCRIPT
![Page 1: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/1.jpg)
Just Enough WebOpsfor Developers
Alexis Lê-Quôc @alqhttp://www.datadoghq.com
![Page 2: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/2.jpg)
@alq
![Page 3: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/3.jpg)
@alq
Co-founder DATADOG
![Page 4: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/4.jpg)
Datadog is Monitoring that does not suck... as a Service
![Page 5: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/5.jpg)
Datadog is Monitoring that does not suck... as a Service
“Metrics made social”
![Page 6: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/6.jpg)
People-friendly Monitoring
![Page 7: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/7.jpg)
Developer-friendly Monitoring
![Page 8: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/8.jpg)
Dev Ops930,000 350,000
2010 US figures from BLS
![Page 9: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/9.jpg)
The New Development
Equation
![Page 10: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/10.jpg)
Code + + AWS =
The New Development Equation
![Page 11: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/11.jpg)
Code + + AWS =3 months
The New Development Equation
![Page 12: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/12.jpg)
Code + + AWS =3 months 5 minutes
The New Development Equation
![Page 13: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/13.jpg)
![Page 14: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/14.jpg)
Web Operations?
![Page 15: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/15.jpg)
Code + + AWS =3 months 5 minutes
The New Development Equation
![Page 16: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/16.jpg)
Code + + AWS =3 months 5 minutes
Web Operations?
The New Development Equation
![Page 17: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/17.jpg)
![Page 18: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/18.jpg)
Cargo cult Operations
![Page 19: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/19.jpg)
![Page 20: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/20.jpg)
Common vocabularybetween Dev & WebOps?
![Page 21: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/21.jpg)
Users
SysAdmin
![Page 22: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/22.jpg)
“Come and get it”
“We want root!”
![Page 23: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/23.jpg)
Dev
WebOps
![Page 24: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/24.jpg)
WebOps
and this is what I do
![Page 25: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/25.jpg)
But first an important digression
![Page 26: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/26.jpg)
Product Service
![Page 27: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/27.jpg)
Service = Code + Infrastructure
![Page 28: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/28.jpg)
Service = Product + Access
![Page 29: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/29.jpg)
![Page 30: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/30.jpg)
Provide access
![Page 31: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/31.jpg)
Provide access
![Page 32: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/32.jpg)
Provide access
reliable, fast, cheap
![Page 33: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/33.jpg)
Provide access
reliable, fast, cheap
![Page 34: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/34.jpg)
Provide access
reliable, fast, cheap
24x7without going crazy
![Page 35: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/35.jpg)
24x7 && !crazy
![Page 36: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/36.jpg)
DevelopmentModels
![Page 37: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/37.jpg)
![Page 38: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/38.jpg)
Delivery historicallynot the focus
![Page 39: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/39.jpg)
Agile Cycle Delivery
![Page 40: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/40.jpg)
Agile Cycle Delivery
![Page 41: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/41.jpg)
Agile Cycle DeliveryWebOps Cycle
![Page 42: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/42.jpg)
WebOps
and this is what I do
![Page 43: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/43.jpg)
Dev Release Measure & Log
Monitor
AlertInvestigate
Change
Fix || Escalate
WebOps Cycle
![Page 44: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/44.jpg)
(Release)
![Page 45: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/45.jpg)
Dev Release
Monitor
AlertInvestigate
Change
Fix || Escalate
Measure & Log
![Page 46: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/46.jpg)
Measure
PurposeCollect quantitative metrics
ProcessInstrument serversInstrument codeInstrument SaaS depsAutomate collection
RisksImprecise metric definitionManual collection“What does it mean?”
ToolsSystem (ganglia, collectd, munin, nagios, etc.)Code (metrics, statsd)SaaS (Datadog et al.)
![Page 47: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/47.jpg)
Dev Release
Monitor
AlertInvestigate
Change
Fix || Escalate
Measure & Log
![Page 48: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/48.jpg)
Log
PurposeCollect meaningful, timestamped events
ProcessAll the timeIn one placeAccess for everyoneDiscipline
RisksTiB of garbageNon-uniform timestampsNon-uniform formats
Toolslog4j et al.syslog et al.logstash, splunk+ Logging-as-a-Service
![Page 49: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/49.jpg)
Dev Release Measure & Log
AlertInvestigate
Change
Fix || Escalate
Monitor
![Page 50: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/50.jpg)
Monitor
PurposeWatch actionable events & metrics
ProcessHealth of the app?Which metrics for health?Compute metricsMetric domainAccess for everyonePretty graphs
RisksNon-actionable metrics
Toolsgraphite, cubism et al.+ services
![Page 51: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/51.jpg)
Dev Release Measure & Log
Monitor
Investigate
Change
Fix || Escalate
Alert
![Page 52: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/52.jpg)
Alert
PurposeBring human in the loopwhen automated fix does not work
ProcessAlert on vital monitorsAdd new alerts with new monitorsCompute metrics from alertsRuthlessly edit
RisksToo many alertsBecome desensitizedIgnore alertsApp crashes for realPendulum swings back
Toolsnagios+ services
![Page 53: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/53.jpg)
Dev Release Measure & Log
Monitor
AlertInvestigate
Change
Fix || Escalate
![Page 54: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/54.jpg)
Fix || Escalate
PurposeFix issue or find someone who can
Process(fix) capture actions as soon as possible (while or shortly after)(fix) runbooks(fix) automate fixes(escalation) on-call rotation(escalation) agree on rules
RisksBurn out
ToolsPagerDutyBug tracker
![Page 55: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/55.jpg)
Dev Release Measure & Log
Monitor
Alert
Change
Fix || Escalate
Investigate
![Page 56: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/56.jpg)
Investigate
PurposeCollect evidenceReconstruct what happened
ProcessStart where/when problem 1st detectedWork your way from thereCapture relevant graphs/logs
RisksMissing the starting pointLagging events/metricsLow-level events/metricsBlame game
ToolsPost-mortems
![Page 57: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/57.jpg)
Dev Release Measure & Log
Monitor
AlertInvestigate
Fix || Escalate
Change
![Page 58: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/58.jpg)
Change
PurposeFewer alertsBetter service
ProcessChange infrastructure, codeInfrastructure == codeAdd/Edit monitors & alerts
Risksad-hoc changes
Tools...
![Page 59: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/59.jpg)
WebOps
and this is what I do
![Page 60: Just enough web ops for web developers](https://reader031.vdocuments.site/reader031/viewer/2022020110/55954fe01a28ab39398b45a5/html5/thumbnails/60.jpg)
Dev Release Measure & Log
Monitor
AlertInvestigate
Change
Fix || Escalate
Questions?Comments?
@alq