root cause analysis - pdated practices for new problems
TRANSCRIPT
a persistent problem
“We also know that the increasing use of larger and more complex systems potentially results in a greater number of problems. Furthermore, many of these problems have a more series impact on the business and on the use of systems than ever before, and they are also in many cases much more difficult to solve.”
--A Management System for the Information Business
2
same RCA, new technologies
• Virtualization
• Multi-tier
• Cloud Computing
3
virtualization
• More to manage - virtualization layer, virtual networks, virtual storage, etc.
• Tracking transients - removing the constraints of physical reality
4
common virtualization
• Correlating virtual instances to physical problems
• Configuration drift
• Network configuration
• VM access management
• Capacity management
5
multi-tier applications
• Dividing monolithic applications
• More parts, more connections to break
• Inter-dependencies cause weird failures
• Cascading problems well suited for RCA
6
common multi-tier problems
• Blizzard of logs, events, an messages
• Memory leaks and resource limit cascade upwards
• Network connection failure
• Storage issues - network drives and shared disks
• Cache updates and gardening
7
cloud computing
• Public vs. private cloud
• Virtualization problems & more
• Managing cloud provider relationship
8
common cloud problems
• Remote access & instrumentation
• Configuration drift, resource degradation, networking, etc.
• Provider escalation policy & process
• Start with “status pages”
9
Root Cause Analysis in
• Relationship tracking
• Event & change tracking
• Tickets & system of record
• Business impact
10
rca metrics
• Mean time to isolate
• SLAs
• Costs - how much does RCA cost, save, make
• Revenue generation
11
go ahead, be a firefighter
12
ContactMichael Coté
512.795.4307
Thank you!http://creativecommons.org/licenses/by-nc-sa/2.5/
License
13
Credits & Co.
14
• Virtualization image: Chen Zhao, http://www.flickr.com/photos/livepine/177132253/
• Russian Dolls: http://www.flickr.com/photos/barteverts/2481199686/
• Typing: http://www.flickr.com/photos/indi/113183027