morgue , helping better understand events by building a post mortem tool - bethany macri
Post on 26-Jan-2015
121 Views
Preview:
DESCRIPTION
TRANSCRIPT
MorgueA Post Mortem Tool
Etsy@BethanyMacri
Monday, October 21, 13
Daniel Schauenberg@mrtazz
github.com/mrtazz
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Shit happens.
Monday, October 21, 13
What I’m talking about today• Post mortems
• Morgue
Monday, October 21, 13
What is a post mortem?
Monday, October 21, 13
Types of events
Monday, October 21, 13
Types of events• Site outage
Monday, October 21, 13
Types of events• Site outage• New users cannot register
Monday, October 21, 13
Types of events• Site outage• New users cannot register• Buyers cannot checkout
Monday, October 21, 13
Types of events• Site outage• New users cannot register• Buyers cannot checkout• Sellers cannot list items
Monday, October 21, 13
Types of events• Site outage• New users cannot register• Buyers cannot checkout• Sellers cannot list items• Data loss
Monday, October 21, 13
Types of events• Site outage• New users cannot register• Buyers cannot checkout• Sellers cannot list items• Data loss• Performance degradation
Monday, October 21, 13
Severity levels
P1 Complete outage
P2 Loss or degradation for subset
P3 Loss of minor functionality
P4 No member-visible impact
P5 Anything else
Monday, October 21, 13
How is a post mortem conducted?
Monday, October 21, 13
Who• The person/people who:
Monday, October 21, 13
Who• The person/people who:
• introduced the problem
Monday, October 21, 13
Who• The person/people who:
• introduced the problem• identified the problem
Monday, October 21, 13
Who• The person/people who:
• introduced the problem• identified the problem• responded to the problem
Monday, October 21, 13
Who• The person/people who:
• introduced the problem• identified the problem• responded to the problem• debugged the problem
Monday, October 21, 13
Who• The person/people who:
• introduced the problem• identified the problem• responded to the problem• debugged the problem• anyone interested
Monday, October 21, 13
What to discuss
Monday, October 21, 13
What to discuss• what happened, and when
Monday, October 21, 13
What to discuss• what happened, and when• assumptions
Monday, October 21, 13
What to discuss• what happened, and when• assumptions• solutions
Monday, October 21, 13
What to discuss• what happened, and when• assumptions• solutions• prevention
Monday, October 21, 13
Monday, October 21, 13
“It’s not your fault”
Monday, October 21, 13
Blameless
Monday, October 21, 13
What to record
Monday, October 21, 13
What to record• Metrics
Monday, October 21, 13
What to record• Metrics• Possible investigations
Monday, October 21, 13
What to record• Metrics• Possible investigations• Possible resolutions
Monday, October 21, 13
What to record• Metrics• Possible investigations• Possible resolutions• Assumptions
Monday, October 21, 13
What to record• Metrics• Possible investigations• Possible resolutions• Assumptions• Remediation
Monday, October 21, 13
So what’s wrong with a wiki?
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
What is Morgue?
Monday, October 21, 13
Stack• PHP
• Slim framework
• MySQL• Apache• Twitter bootstrap
Monday, October 21, 13
Current features• General • Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Severity level
• Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Severity level
• Timezone support
• Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• Etsy
Monday, October 21, 13
Current features• General
• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
• Etsy
Monday, October 21, 13
Current features• Etsy• General
• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
Monday, October 21, 13
Current features• Etsy
• Etsystatus time
• General• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
Monday, October 21, 13
Current features• Etsy
• Etsystatus time
• Contact/GCal
• General• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
Monday, October 21, 13
Current features• Etsy
• Etsystatus time
• Contact/GCal
• IRC
• General• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
Monday, October 21, 13
Current features• Etsy
• Etsystatus time
• Contact/GCal
• IRC
• Jira
• General• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
Monday, October 21, 13
Current features• Etsy
• Etsystatus time
• Contact/GCal
• IRC
• Jira
• Forums
• General• Time calculations
• Severity level
• Timezone support
• Markdown for “What Happened”
• Images
• Tags
• History
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
10/19/2013
Monday, October 21, 13
10/19/2013
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
How did Morgue help?
Monday, October 21, 13
Severity levels recorded
0
25
50
75
100
P1 P2 P3 P4 P5 No data
Wiki
Monday, October 21, 13
Severity levels recorded
0
25
50
75
100
P1 P2 P3 P4 P5 No data
Wiki Morgue
Monday, October 21, 13
0
25
50
75
100
Severity Times IRC Remediation Images
Wiki
Monday, October 21, 13
0
25
50
75
100
Severity Times IRC Remediation Images
Wiki Morgue
Monday, October 21, 13
What’s the future of Morgue?
Monday, October 21, 13
Feature ideas• Graphs
• by severity• by tags
• Remediation enforcement• Copy to warn “should”s and “could”s
Monday, October 21, 13
github.com/Etsy/morgue
Monday, October 21, 13
Pull requestswelcome
Monday, October 21, 13
Questions?github.com/Etsy/morgue
@BethanyMacribmacri@etsy.com
Monday, October 21, 13
Source Credits http://media.heavy.com/media/2013/03/morgue.jpg
http://authorstephanihecht.files.wordpress.com/2012/11/mistake-2.jpg?w=640
http://4.bp.blogspot.com/-A9OibuJMGlM/TfSyX6cMjXI/AAAAAAAAADc/cM4JwCkt1zg/s1600/Good.Will.Hunting.1997.XviD.DTS.CD3-WAF.avi_001529070.jpg
https://www.hackerschool.com/assets/slideshow_01-540531c0d5690f948a4d5d02c5400827.jpg
http://twa2r.files.wordpress.com/2011/01/neo-bullets.jpg
http://commons.wikimedia.org/wiki/File:University-of-Virginia-Rotunda.jpg
http://abovethelaw.com/wp-content/uploads/2012/03/law-school-gavels.jpg
http://nyogalleristny.files.wordpress.com/2012/05/8850-munch-the-scream-e1336013995741.jpg?w=300&h=233
http://smallbizelevator.com/wp-content/uploads/et_temp/stay-handmade-etsy-63270_200x200.jpg
Monday, October 21, 13
Monday, October 21, 13
top related