Upload File

incident management and chatops - usenix forget the incident response procedure spy pre-oncall...

  • Home
  • Documents
  • Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $
28
Incident Management and ChatOps ❤ Daniella Niyonkuru (@niyodanie)

Upload: phamtram

Post on 27-Mar-2018

252 views

Category:

Documents


4 download

Report

TRANSCRIPT

Page 1: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Incident Management and ChatOps❤🔥💬🤖

Daniella Niyonkuru (@niyodanie)

Page 2: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Production Engineering

Page 3: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

🔥

Page 4: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

I N C I D E N T M A N A G E R O N - C A L L ( I M O C )

Page 5: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

IMOCSupport Response Manager

(SRM)

Component Experts

Incident Command System (ICS)

Page 6: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

The IMOC is on-call for Incident Response;

NOT on-duty for fixing production issues.

Page 7: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Incident Response Funnel➡ Shit breaks ➡ Detection ➡ Start Incident ➡ Communicate ➡ Fix ➡ Stop Incident ➡ Document (Service Disruption) ➡ Investigation ➡ Root Cause Analysis (RCA) ➡ Action Items ➡ ResolutionCredit: John Arthorne

Page 8: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Pager Anxiety; What if …

• Forget I’m on-call

• Phone in silent mode

• Forget to update the status page

• Don’t know who to ping

• Too much context switching, can't focus

• Forget the incident response procedure

😱

Page 9: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

C H ATO P S

Page 10: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Conversation-Driven Collaboration

Page 11: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Chatops at Shopify

Page 12: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

I N C I D E N T S A N D C H ATO P S

Page 13: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Incident Management Tools

Three main sets of commands:

• spy page

• spy incident

• spy status

Page 14: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Shit breaks➡ spy page imoc “order notifications not going out”

Page 15: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Start incident➡ spy incident start me order fraud analysis outage

Page 16: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Communicate➡ spy incident tldr

Page 17: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Other Teams➡ spy incident tell :team message➡ spy page datastores

Page 18: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Actions

Page 19: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Third-Party Services➡ spy status➡ spy status :provider :status for :feature➡ spy pager imoc res 123

Page 20: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Reminders when: [30, stop] command: :check_status_page- when: 120 command: :notify_support_atc message: 'Spy has notified the Support Response Manager (SRM) on your behalf.'- when: 120 command: :srm_fill_out_doc- when: 300 message: 'You should coordinate external comms with the support incident responder.’- when: 600 command: :srm_checking_in- when: [3600] command: :notify_imoc_team- when: stop message: 'Please create a Service Disruptions report.' Milestones

Page 21: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Stop incident➡ spy incident stop

Page 22: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

And much more

• SD content generation (`spy incident note`)

• Preventing on-call fatigue (`spy incident handoff`)

• Reducing context switching (`spy pager stfu`)

• Reminders (before, during and after the incident)

Page 23: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

H O W D I D S P Y A F F E C T I M O C S ?

Page 24: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Benefits

• Increased sharing and focus

• Shortened feedback loop

• Eliminated manual toil

• Smoother incident handling

• Faster onboarding experience

Page 25: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Fears, What if …• Forget I’m on-call

• Phone in silent mode

• Forget to update the status page

• Don’t know who to ping

• Too much context switching, can't focus

• Forget the incident response procedure

spy pre-oncall reminders

spy check reminders

spy oncall

spy cmd #war-room

spy incident

Page 26: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

🤖

• Flexible and powerful

• A very important member of our team

• Enables us to really lead an incident response

• Reduce incident impact and duration

Page 27: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Questions?

THANK YOU!

@niyodanie

Page 28: Incident Management and ChatOps - USENIX Forget the incident response procedure spy pre-oncall reminders spy check reminders spy oncall spy cmd #war-room spy incident $

Shopify TalksThursday5:00 pm to 6:00 pm:Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) - Jaime Woo

Friday11:30 am to 12:00 pm: Building an On-Premise Kubernetes Cluster For a Large Web Application - Daniel Turner

Check out our blog at engineering.shopify.comFollow us on Twitter at @shopifyeng


MIC3562 OnCall Trade Ad 165x65mm AW

U-2 Spy Incident (1960) Col. Francis Gary Powers plane was shot down over Soviet airspace

เวรsknhospital.go.th/fileTDoctor/23.pdf · 2014-06-02 · ตารางเวรแพทย รอรับปรึกษา Oncall นอกเวลาราชการ

AUSTRALIAN WORKPLACE AGREEMENT FOR KERANG/media/health/files... · Web viewChange of Shift Allowance Y 2611 Telephone OnCall Y 2612 Telephone OnCall PH Y 2615 Early Morning Shift

U-2 Spy Incident (1960)

Suga BURKE FINE SEASHELL TRANSFER POPSICLE THEMED I SPY GAME spy: POPSICLES spy … · 2018-05-22 · POPSICLE THEMED I SPY GAME spy: POPSICLES spy: POPSICLES Collages Sponge Bombs

Spy or Rebel? The Curious Incident of the Temurid Sultan ...mamluk.uchicago.edu/MSR_XIV_2010-Broadbridge-pp29-42.pdfThe Curious Incident of the Temürid Sulṭān-Ḥusayn’s Defection

Spy v Spy - Treachery in the Dev/Ops Trenches

2015 OnCall Quarter 1

I spy, you spy - Jacaranda · I spy, you spy spy noun 1. an agent employed by a state to obtain secret information, especially of a military nature, concerning its potential or actual

Spy tape versus spy tape

On Her Majesty's Secret Service - GRX & A spy agencyconference.hitb.org/hitbsecconf2014ams/materials/D2T1-GRX-and-a … · • KPN-CERT & REDteam • Penetration Testing & Incident

Hd --720 spy camera spy carkey camera

Jadwal Oncall September

SPY-DVR4 / SPY-DVR8 / SPY-DVR16 Instructionsmetrahometheater.com/media/files/SPY-DVRmanual.pdf · 2015. 10. 19. · •Color Adjust: To adjust Hue, Brightness, Saturation, and Contrast

Dexcomm How to -Web OnCall

Handbook From Obstacles to Opportunities24 AccessPay Salary Packaging 25 Nadrasca 26 Level Consulting Services 27 ONCALL Training College 28 ONCALL Group 29 Toustone 34 Every Australian

TENNESSEE BANKERS ASSOCIATION STRATEGIC Strategic Tech... · Phoenix, Spy-Eye, and Citadel have ... Cybersecurity, Third-Party Management, Incident Response, and Other Key Topics

Wiley.verilog.coding.for.Logic.synthesis.ebook Spy

Welcome to OnCall Benefits. The benefits you need, when you need them

Spy camera

PRINT POSTERS AT 100% A Simulated First Night-onCall (FNOC) · PRINT POSTERS AT 100% A Simulated First Night-onCall (FNOC): Establishing Community and a Culture of Patient Safety

spy in-plant holiday detectors - SPY Pipeline Inspection

Reflections: Neurology and the Humanities Oncall Carl

SES San Francisco: Spy vs. Spy - Competitive Research

SPY-DVR16HYB2 & SPY-DVR16HYB2ND - Metra Home …metrahometheater.com/media/files/SPY-DVR16HYB2 MANUAL (2).pdf · SPY-DVR16HYB2 & SPY -DVR16HYB2ND ... • Up to 16 camera capacity

HxGN OnCall Field Mobility Solutions

enVision - Anadolu Üniversitesi · spy 255 masa tenis1 (t) spy 253 tenis (t) spy 255 masa tenisi (u) spy 253 spy 255 masa tenisl spy 253 (u) ana 126 kinesiyolojik anatomi (t) ana

SPY LYMPHATICS KIT- indocyanine green SPY-MIS KIT

Spy vs Spy: Protecting Secrets

GAME-SPY through GACP - Cisco...GAME-SPY Name/CLI Keyword game-spy Full Name game-spy Description Game-spy online gaming Reference - Global ID L4:1349 ID 1349 Known Mappings UDP Port

TÀI LIỆU MÔ TẢ API VÀ EVENT URL ONCALL - FTI · 2020. 12. 1. · FPT TELECOM INTERNATIONAL HOTLINE: 1900 6973 3 TÀI LIỆU MÔ TẢ API VÀ EVENT URL ONCALL - Output: Trường

Spy and Counter Spy

Spy vs Spy Score

MANUAL PENGGUNA SISTEM ONCALLoncall.ukm.my/files/manual/penyelaras-jadual.pdfManual Pengguna Sistem Oncall – Penyelaras Jadual 5 MENU PENYELARAS JADUAL 1. Penjanaan Jadual Oncall