it problems & problem management
TRANSCRIPT
![Page 1: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/1.jpg)
![Page 2: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/2.jpg)
Session Date & Time• Date: Wednesday, June 29, 2016
Time: 1100-1145Location: Tuscany Ballroom
![Page 3: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/3.jpg)
• Bill Alderson responds to Information Technology high visibility, high stakes technical problems. Network outage, slowness, slow applications or disasters affecting government and commercial Information Technology Enterprise environments. ABC News told the story of how Bill and his team helped restore communications at the Pentagon immediately following 911. Bill assisted with six deployments to Iraq and Afghanistan requested by Army G2, Joint Chiefs and US Central Command diagnosing Biometrics and others critical systems. One of his missions is to help executives and technologists see both technical and leadership root causes that can be obviated through common sense best practices.
![Page 4: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/4.jpg)
Bill Alderson Infographic Bio
![Page 5: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/5.jpg)
• Deep packet analysis remains essential for definitive irrefutable diagnosis and optimization of complex systems. Bill demonstrates the tools, techniques and methods used to annotate complex technology findings so that technologists, managers, executives and vendors can agree on root cause. Once the problem is identified and agreed upon the true pinpoint mitigation can begin. The days of shotgun style "forklift wholesale upgrades" on everything have passed. We must optimize existing assets allowing them to perform well.
Bill has proven ability to optimize large scale networks and applications from experience in analyzing the Pentagon immediately following 911, analysis of Biometrics applications across Iraq and Afghanistan, numerous optimizations of Joint Chiefs of Staff and OSD network analysis. Experience from analysis of the largest 100 commercial enterprise networks such as Stock Exchanges, Financial, Insurance and Healthcare institutions will be demonstrated with annotated examples for CIO, Executives and top level technologists.
![Page 7: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/7.jpg)
“Swiss Army Knife” Portfolio of Tools
Select Well.Avoid SpendingOnly on “Suites”
All-in-one-toolsAlthough easier to “buy”
don’t solve many problems.They leave you “broke and broken”
with a gold plated toolset.
![Page 8: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/8.jpg)
Optimization Troubleshooting Phases
![Page 9: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/9.jpg)
Preparation & Setup
![Page 10: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/10.jpg)
Analysis & Iteration
![Page 11: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/11.jpg)
Reporting & Presentation
![Page 12: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/12.jpg)
Problem Management
![Page 13: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/13.jpg)
Down - Intermittent - Slow
![Page 14: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/14.jpg)
Technical vs. Leadership Root Cause
![Page 15: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/15.jpg)
The Needle
![Page 16: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/16.jpg)
The Environment
![Page 17: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/17.jpg)
Packet Traces
![Page 18: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/18.jpg)
Store Every Packet? Who’s can and is going to analyze them and when?
![Page 19: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/19.jpg)
Finding The Stack With The Problem
![Page 20: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/20.jpg)
Finding The Needle
![Page 21: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/21.jpg)
Measured at the Server
Fast TCP connect time. Fast Ack from F5 does not show true client response time which is why Apalytics provided Internet Monitoring.
1.4 second Get response is very slow which is why detailed platform and application analysis was performed.
The 2nd & 3rd Gets were fast at 1 millisecond proving some commands are fast.
![Page 22: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/22.jpg)
CF Longest Requests
1,958,266ms = ~32 minutes from one request391,692ms = ~7 minutes
![Page 23: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/23.jpg)
Page Analysis from the Internet DNS does not play a role in slowness. Connection time varies and at time approaches 200 milliseconds which can be at the platform, internet, network, load balancer or firewalls. Connection delay analysis will require multiple capture points to definitively pinpoint and should be considered when multi-point capture test points can be configured at the Security Tap devices. But that is not material for improvement of this application at this timeFirst byte time is the most concerning issue in the infrastructure. Last byte time is also a concern as it appears that platform TCP/IP stack services are slow to move data out onto the wire after the first byte has started. It may also be that platform improvements may improve both response times and output speed. Page load time is a composite of all elements of the page that must come together to provide the user with the visual page and the main context of the query. This too is concerning, but it is caused by the slowness of the individual components of the page as they add serially to the response time which are represented in the main concerns. An example of the total page would be small visual images and data making up the user interface view (i.e., logos) that are not part of a computational or lookup, but rather a static image that should be served rapidly by the server.
![Page 24: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/24.jpg)
Network Intrinsic Application Analysis
![Page 25: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/25.jpg)
Multi-tier Analysis
![Page 26: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/26.jpg)
Multi-Tier Identification
![Page 27: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/27.jpg)
Application Monitoring Design Phase
![Page 28: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/28.jpg)
Multi-tier Macro vs. Micro
Event
Process
Net-Ser-Tr-Sw-Q
Security Auth
User ClickClientNetworkWebSvrNetworkAppSrvNetworkSQLSvrNetworkAppSvrNetworkMainframeNetworkAppSvrNetworkWebSvrNetworkClientUser Display UpdateMacro Response
Time
Micro Response Time
![Page 29: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/29.jpg)
HTTP Post from client
Web1 Middlewa
re 155ms
HTTP / SQL Multi-tier 1
![Page 30: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/30.jpg)
Back to clientWith HTTP
SQL Calls completeQuery and returns Rows to Web1
SQL Calls finish .497SQL Call start -.231
SQL Resp Time =.266
Web1 Middleware
12ms
HTTP / SQL Multi-tier 2
![Page 31: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/31.jpg)
Logon A is 72 milliseconds…
Logon B is 420 milliseconds!
Oracle Logon Slow
![Page 32: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/32.jpg)
Micro-Analysis Phase
Web App I/F #1&2 SQL TransLogger MF#1 MF#2 Time Breakdown
![Page 33: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/33.jpg)
TCP Satellite Retrans 3.5 Seconds
![Page 34: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/34.jpg)
Processing Analysis
![Page 35: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/35.jpg)
Packet Loss Analysis
![Page 36: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/36.jpg)
Citrix Session Abort Signature “Chernobyl Packet”
The packet that evidenced a problem on a Citrix server. This pattern was used as a signature on the Infinistream Sniffers to find these problems until they were remediated.
Prior to this users were stuck in this cycle for hours.
![Page 37: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/37.jpg)
Citrix User Filer Access Error Details
![Page 38: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/38.jpg)
Blind vs. Pinpoint Upgrades
Blind Upgrade = Shotgun Approach = Forklift Upgrade
![Page 39: IT Problems & Problem Management](https://reader034.vdocuments.site/reader034/viewer/2022051504/58eda8491a28ab631b8b4689/html5/thumbnails/39.jpg)
Root Cause Optimization
Definitive Root Cause Analysis Pinpoint Cause Measure ROI PotentialPinpoint Purchases Validate & Prove ROI Award Innovation
OptimizationRoot
Cause Analysis