Advanced Troubleshooting Advanced Troubleshooting with Debug Diagnostics on with Debug Diagnostics on IIS 6IIS 6

Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product UnitMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

Debug Diagnostics 1.0Debug Diagnostics 1.0

Dig into crash scenariosDig into crash scenarios

Understanding hangsUnderstanding hangs

Investigating memory leaksInvestigating memory leaks

Session SummarySession Summary

Debug Diagnostics 1.0Debug Diagnostics 1.0““Core” pieces of DebugDiagCore” pieces of DebugDiag

Understanding “Rules”Understanding “Rules”

Using DebugDiag’s User Using DebugDiag’s User InterfaceInterface

Core pieces of Debug DiagCore pieces of Debug Diag

3 pieces make up Debug Diag 1.03 pieces make up Debug Diag 1.0ServiceService

Host (also called Controller)Host (also called Controller)

User InterfaceUser Interface

DbgSvc.exeDbgSvc.exe DbgHost.exeDbgHost.exe DebugDiag.exeDebugDiag.exe

Core pieces of Debug DiagCore pieces of Debug Diag

Service is multi-Service is multi-purposedpurposed

Interacts with the Interacts with the Debugger HostDebugger Host

Allows:Allows:Service required to Service required to attach invasively to attach invasively to processprocess

To run from To run from Terminal ServicesTerminal Services

The Service – The Service – DbgSvc.exeDbgSvc.exe

DbgSvc.exeDbgSvc.exe

To register the service:To register the service:C:\DebugDiag> DbgSvc /serviceC:\DebugDiag> DbgSvc /service

To start the service:To start the service:C:\DebugDiag> net start DbgSvcC:\DebugDiag> net start DbgSvc

To stop the serviceTo stop the serviceC:\DebugDiag> net stop DbgSvcC:\DebugDiag> net stop DbgSvc

Unregister the serviceUnregister the serviceC:\DebugDiag> DbgSvc /unregserverC:\DebugDiag> DbgSvc /unregserver

Core pieces of Debug DiagCore pieces of Debug Diag

HostHostWhere the Where the DbgEng.dll DbgEng.dll instance is loadedinstance is loaded

Exposes 3 classesExposes 3 classes

The Host – DbgHost.exeThe Host – DbgHost.exe

DbgControlDbgControl:: Attach/Detach from processes or Attach/Detach from processes or open/analyze a open/analyze a

memory dumpmemory dump

DbgObjDbgObj:: Collect process or memory dump Collect process or memory dump informationinformation

ManagerManager:: Mainly to output analysis data to Mainly to output analysis data to the the report filereport file

DbgSvc.exeDbgSvc.exe

DbgHost.exeDbgHost.exe

AttacAttachh

Core pieces of Debug DiagCore pieces of Debug Diag

User Interface User Interface designed to designed to simplify:simplify:

Rule creationRule creation

View processessView processess

Initiate analysis of Initiate analysis of memory dumpsmemory dumps

Add analysis Add analysis scriptsscripts

Change properties Change properties for Debug Diagfor Debug Diag

The User Interface – The User Interface – DebugDiag.exeDebugDiag.exe

DebugDiag.exeDebugDiag.exe

DbgHost.exeDbgHost.exe

Control Control ScriptsScripts

Crash ScenariosCrash Scenarios

Background of crashesBackground of crashes

Using DebugDiag to Using DebugDiag to quickly identify root quickly identify root cause of crashescause of crashes

Background: ExceptionsBackground: Exceptions

What is an What is an exception?exception?

How do applications How do applications cause exceptionscause exceptionsWhat is a “handled” What is a “handled” exception?exception?Unhandled Unhandled exceptions need exceptions need debugging and debugging and resolutionresolution

Access Violation: Access Violation: C000 0005C000 0005

Breakpoint Exception: Breakpoint Exception: 8000 00038000 0003

Stack Overflow: Stack Overflow: C000 00FDC000 00FD

C++ exception: C++ exception: E06D7363E06D7363

Managed exception: Managed exception: E0434F4DE0434F4D

Invalid Handle: Invalid Handle: C000 0008C000 0008

Background: Heap (2)Background: Heap (2)What is a heap?What is a heap?

Heap is a data structureHeap is a data structure

Heap Corruption: Heap Corruption: Certain rules must be followed when using the Certain rules must be followed when using the heapheap

When rules are violated, heap corruption When rules are violated, heap corruption occursoccurs

Heap corruption Heap corruption crashescrashes

Code that corrupts heap is usually not the ones Code that corrupts heap is usually not the ones impactedimpacted

Debugging CrashesDebugging Crasheswith Debug Diagnostics:with Debug Diagnostics:Crash RuleCrash Rule

Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit

Hang ScenariosHang Scenarios

Investigating the Investigating the theory of theory of “process hangs”“process hangs”

Using DebugDiag to Using DebugDiag to debug “process debug “process hangs”hangs”

Theory of “Process Hangs”Theory of “Process Hangs”

What is a process?What is a process?Memory boundary for execution of codeMemory boundary for execution of code

Allocated by Operating System done by calling Allocated by Operating System done by calling CreateProcess or CreateProcessAsUser APICreateProcess or CreateProcessAsUser API

Processes do not execute codeProcesses do not execute code

Contains a minimum of 1 threadContains a minimum of 1 thread

What is a thread?What is a thread?An independent sequence of instructions An independent sequence of instructions executing within a processexecuting within a process

Theory of “Process Hangs”Theory of “Process Hangs”

Thread implications on applicationsThread implications on applicationsThe level of concurrency (virtual) is dictated by The level of concurrency (virtual) is dictated by the number of threadsthe number of threads

Real concurrency is dictated by the number of Real concurrency is dictated by the number of execution unitsexecution units

1 processor = one or more execution units1 processor = one or more execution units

What are What are common thread common thread

pools for IIS?pools for IIS?

ATQ thread poolATQ thread poolASP thread poolASP thread poolASP.NET thread poolASP.NET thread pool

Runaway ThreadsRunaway Threads

A runaway thread is a thread stuck in a A runaway thread is a thread stuck in a infinite (or near) loopinfinite (or near) loop

Performing a task that is consuming CPUPerforming a task that is consuming CPU

Either by design or because of error conditionsEither by design or because of error conditions

Runaway thread is taking CPU resources Runaway thread is taking CPU resources away from other threadsaway from other threads

Often takes away from good threadsOften takes away from good threads

Causes performance degradation for server Causes performance degradation for server

SQLThreadThread44

ThreadThread33

ThreadThread11

Defining a Blocked ThreadDefining a Blocked Thread

A thread that isn’t using any CPU resourcesA thread that isn’t using any CPU resources

Waiting on database calls to returnWaiting on database calls to return

Waiting on web service calls to returnWaiting on web service calls to return

Waiting on a socket/network call to returnWaiting on a socket/network call to return

Waiting on a lockWaiting on a lock

IIS

Locks and DeadlocksLocks and DeadlocksWhat is a lock?What is a lock?

A synchronization mechanism to protect resources from A synchronization mechanism to protect resources from being corrupted due to simultaneous access by multiple being corrupted due to simultaneous access by multiple threadsthreads

Why do developers use locks?Why do developers use locks?Protecting dataProtecting dataProtecting other resources that are not Protecting other resources that are not memory basedmemory based

Impact on applications if locks are Impact on applications if locks are poorly implementedpoorly implemented

Causes threads to blockCauses threads to blockCan cause deadlock situations when using multiple Can cause deadlock situations when using multiple locks in the applicationlocks in the applicationProcess hangs when no threads are availableProcess hangs when no threads are available

Debugging Deadlocks and Debugging Deadlocks and Process Hangs with Process Hangs with DebugDiag: Hang RuleDebugDiag: Hang Rule

Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit

Looking into Leaks…Looking into Leaks…

What is a leak?What is a leak?

Types of LeaksTypes of Leaks

Debugging a Leak Debugging a Leak with Debug with Debug DiagnosticsDiagnostics

Leak Scenarios (2)Leak Scenarios (2)

Why are leaks difficult to identify and Why are leaks difficult to identify and resolve?resolve?

Server applications often cache memory Server applications often cache memory allocationsallocations

The allocations are long termThe allocations are long term

These applications, to outsiders, look to have These applications, to outsiders, look to have memory leaksmemory leaks

There are many other cachesThere are many other cachesUsing Performance Monitor and seeing Using Performance Monitor and seeing memory growth does not equal a memory leakmemory growth does not equal a memory leak

Leak Scenarios (3)Leak Scenarios (3)

What are short-term allocations?What are short-term allocations?Memory expensive applications use “spike” Memory expensive applications use “spike” allocations but are destroyed immediately after allocations but are destroyed immediately after request (not long- sustaining)request (not long- sustaining)

These are These are notnot Memory Leaks Memory Leaks

Leak Scenarios (4)Leak Scenarios (4)

Difference between cache and memory Difference between cache and memory leaks?leaks?

Memory leaks are caused by unbounded Memory leaks are caused by unbounded allocationsallocations

Cache memory usage rises quickly, but tops Cache memory usage rises quickly, but tops out after application is stabilizedout after application is stabilized

Nailing a Memory Leak in a Nailing a Memory Leak in a Web Application using Web Application using DebugDiag: Memory RuleDebugDiag: Memory Rule

Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit

Session SummarySession SummaryDebugging Live Web Applications without downtime is Debugging Live Web Applications without downtime is challengingchallenging

Crashes cause processes to terminate due to unhandled Crashes cause processes to terminate due to unhandled exceptions exceptions

Hangs are caused due to race conditions, lock contention, Hangs are caused due to race conditions, lock contention, network blocking, database blocking, infinite loops and network blocking, database blocking, infinite loops and deadlocksdeadlocks

Leaks are caused by the failure to release memory Leaks are caused by the failure to release memory allocated beyond typical short term allocations and allocated beyond typical short term allocations and caching caching

Debug Diagnostics has an extensibility model that allows Debug Diagnostics has an extensibility model that allows it to successfully capture and analyze common debugging it to successfully capture and analyze common debugging scenarios to determine the root causescenarios to determine the root cause

Debug Diagnostics 1.0 web Debug Diagnostics 1.0 web resourcesresources

http://www.iisdiagnostics.comhttp://www.iisdiagnostics.com

© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.