Advanced Troubleshooting Advanced Troubleshooting with Debug Diagnostics on with Debug Diagnostics on IIS 6IIS 6Draft 2.5 5/13/06Draft 2.5 5/13/06

NameNameTitleTitleGroupGroupMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

IntroductionIntroduction

A look at Debug A look at Debug DiagnosticsDiagnostics

Dig into crash scenariosDig into crash scenarios

Understanding hangsUnderstanding hangs

Investigating memory leaksInvestigating memory leaks

Session SummarySession Summary

IntroductionIntroduction

Overview of Debugging: Overview of Debugging: Past and PresentPast and Present

Integral tools for resolving Integral tools for resolving Debug scenariosDebug scenarios

Variables impacting Variables impacting DebuggingDebugging

Overview of DebuggingOverview of DebuggingPast and PresentPast and Present

Past: offered many tools, fewer resolutionsPast: offered many tools, fewer resolutionsPrevious tools were often not server-readyPrevious tools were often not server-readyNo single tool to address different debugging scenariosNo single tool to address different debugging scenariosRequired expertise in debuggingRequired expertise in debugging

Present: more advanced tools, still time-Present: more advanced tools, still time-consuming to solveconsuming to solve

Still lacks any analytical pieceStill lacks any analytical pieceExisting tools are not “integrated” into single toolExisting tools are not “integrated” into single toolDo you really want to learn “debugging?”Do you really want to learn “debugging?”

Talking about tools…Talking about tools…

There are several tools designed at There are several tools designed at resolving the same problemsresolving the same problems

As Windows Servers have grown, so As Windows Servers have grown, so have the debugging toolshave the debugging tools

Example of debugging toolsExample of debugging toolsException MonitorException Monitor

ADPlus.vbs (i.e. – cdb.exe)ADPlus.vbs (i.e. – cdb.exe)

IIS Debug Toolkit 1.1IIS Debug Toolkit 1.1

IIS StateIIS State

Variables Impacting Variables Impacting Debugging (2)Debugging (2)

IIS’s Extensibility ExamplesIIS’s Extensibility ExamplesISAPIISAPI

ASP.NETASP.NET

Active Server Pages (ASP)Active Server Pages (ASP)

COM componentsCOM components

PHP (ISAPI)PHP (ISAPI)

Perl (CGI or ISAPI)Perl (CGI or ISAPI)

Oracle\MySQLOracle\MySQL

These “extensible” parts are often These “extensible” parts are often integrated and executing in same processintegrated and executing in same process

Ever felt like this?

Debug Diagnostics 1.0Debug Diagnostics 1.0““Core” pieces of DebugDiagCore” pieces of DebugDiag

Understanding “Rules”Understanding “Rules”

Using DebugDiag’s User Using DebugDiag’s User InterfaceInterface

Core pieces of Debug DiagCore pieces of Debug Diag

3 pieces make up Debug Diag 1.03 pieces make up Debug Diag 1.0ServiceService

Host (also called Controller)Host (also called Controller)

User InterfaceUser Interface

DbgSvc.exeDbgSvc.exe DbgSvc.exeDbgSvc.exe DebugDiag.exeDebugDiag.exe

Core pieces of Debug DiagCore pieces of Debug Diag

Service is multi-Service is multi-purposedpurposed

Interacts with the Interacts with the Debugger HostDebugger Host

Allows:Allows:Service required to Service required to attach invasively to attach invasively to processprocess

To run from To run from Terminal ServicesTerminal Services

The Service – The Service – DbgSvc.exeDbgSvc.exe

DbgSvc.exeDbgSvc.exe

To register the service:To register the service:C:\DebugDiag> DbgSvc /serviceC:\DebugDiag> DbgSvc /service

To start the service:To start the service:C:\DebugDiag> net start DbgSvcC:\DebugDiag> net start DbgSvc

To stop the serviceTo stop the serviceC:\DebugDiag> net stop DbgSvcC:\DebugDiag> net stop DbgSvc

Unregister the serviceUnregister the serviceC:\DebugDiag> DbgSvc /unregserverC:\DebugDiag> DbgSvc /unregserver

Core pieces of Debug DiagCore pieces of Debug Diag

HostHostWhere the Where the DbgEng.dll DbgEng.dll instance is loadedinstance is loaded

Exposes 3 classesExposes 3 classes

The Host – DbgHost.exeThe Host – DbgHost.exe

DbgControlDbgControl:: Attach/Detach from processes or Attach/Detach from processes or open/analyze a open/analyze a

memory dumpmemory dump

DbgObjDbgObj:: Collect process or memory dump Collect process or memory dump informationinformation

ManagerManager:: Mainly to output analysis data to Mainly to output analysis data to the the report filereport file

DbgSvc.exeDbgSvc.exe

DbgHost.exeDbgHost.exe

AttacAttachh

Core pieces of Debug DiagCore pieces of Debug Diag

User Interface User Interface designed to designed to simplify:simplify:

Rule creationRule creation

View processessView processess

Initiate analysis of Initiate analysis of memory dumpsmemory dumps

Add analysis Add analysis scriptsscripts

Change properties Change properties for Debug Diagfor Debug Diag

The User Interface – The User Interface – DebugDiag.exeDebugDiag.exe

DebugDiag.exeDebugDiag.exe

DbgHost.exeDbgHost.exe

Control Control ScriptsScripts

Crash ScenariosCrash ScenariosBackground of crashesBackground of crashes

Using DebugDiag to Using DebugDiag to quickly identify root quickly identify root cause of crashescause of crashes

Background: ExceptionsBackground: Exceptions

What is an What is an exception?exception?

How do applications How do applications cause exceptionscause exceptionsWhat is a “handled” What is a “handled” exception?exception?Unhandled Unhandled exceptions need exceptions need debugging and debugging and resolutionresolution

Access Violation: Access Violation: C000 0005C000 0005

Breakpoint Exception: Breakpoint Exception: 8000 00038000 0003

Stack Overflow: Stack Overflow: C000 00FDC000 00FD

C++ exception: C++ exception: E06D7363E06D7363

Managed exception: Managed exception: E0434F4DE0434F4D

Invalid Handle: Invalid Handle: C000 0008C000 0008

Background: Heap (2)Background: Heap (2)What is a heap?What is a heap?

Heap is a data structureHeap is a data structure

Heap Corruption: Heap Corruption: Certain rules must be followed when using the Certain rules must be followed when using the heapheap

When rules are violated, heap corruption When rules are violated, heap corruption occursoccurs

Heap corruption Heap corruption crashescrashes

Code that corrupts heap is usually not the ones Code that corrupts heap is usually not the ones impactedimpacted

Debugging CrashesDebugging Crasheswith Debug Diagnostics:with Debug Diagnostics: Crash Rule Crash Rule

NameNameTitleTitleGroupGroup

Hang ScenariosHang Scenarios

Investigating the Investigating the theory of theory of “process hangs”“process hangs”

Using DebugDiag to Using DebugDiag to debug “process debug “process hangs”hangs”

Theory of “Process Hangs”Theory of “Process Hangs”

What is a process?What is a process?Memory boundary for execution of codeMemory boundary for execution of code

Allocated by Operating System done by calling Allocated by Operating System done by calling CreateProcess or CreateProcessAsUser APICreateProcess or CreateProcessAsUser API

Processes do not execute codeProcesses do not execute code

Contains a minimum of 1 threadContains a minimum of 1 thread

What is a thread?What is a thread?An independent sequence of instructions An independent sequence of instructions executing within a processexecuting within a process

Theory of “Process Hangs”Theory of “Process Hangs”

Thread implications on applicationsThread implications on applicationsThe level of concurrency (virtual) is dictated by The level of concurrency (virtual) is dictated by the number of threadsthe number of threads

Real concurrency is dictated by the number of Real concurrency is dictated by the number of execution unitsexecution units

1 processor = one or more execution units1 processor = one or more execution unitsWhat are What are

common thread common thread pools for IIS?pools for IIS?

ATQ thread poolATQ thread poolASP thread poolASP thread poolASP.NET thread poolASP.NET thread pool

Runaway ThreadsRunaway Threads

A runaway thread is a thread stuck in a A runaway thread is a thread stuck in a infinite (or near) loopinfinite (or near) loop

Performing a task that is consuming CPUPerforming a task that is consuming CPU

Either by design or because of error conditionsEither by design or because of error conditions

Runaway thread is taking CPU resources Runaway thread is taking CPU resources away from other threadsaway from other threads

Often takes away from good threadsOften takes away from good threads

Causes performance degradation for server Causes performance degradation for server

SQLThreadThread44

ThreadThread33

ThreadThread11

Defining a Blocked ThreadDefining a Blocked Thread

A thread that isn’t using any CPU resourcesA thread that isn’t using any CPU resources

Waiting on database calls to returnWaiting on database calls to return

Waiting on web service calls to returnWaiting on web service calls to return

Waiting on a socket/network call to returnWaiting on a socket/network call to return

Waiting on a lockWaiting on a lock

IIS

Locks and DeadlocksLocks and DeadlocksWhat is a lock?What is a lock?

A synchronization mechanism to protect resources A synchronization mechanism to protect resources from being corrupted due to simultaneous access by from being corrupted due to simultaneous access by multiple threadsmultiple threads

Why do developers use locks?Why do developers use locks?Protecting dataProtecting dataProtecting other resources that are not Protecting other resources that are not memory basedmemory based

Impact on applications if locks are Impact on applications if locks are poorly implementedpoorly implemented

Causes threads to blockCauses threads to blockCan cause deadlock situations when using multiple Can cause deadlock situations when using multiple locks in the applicationlocks in the applicationProcess hangs when no threads are availableProcess hangs when no threads are available

Debugging Deadlocks and Debugging Deadlocks and Process Hangs with Process Hangs with DebugDiag: Hang RuleDebugDiag: Hang Rule

NameNameTitleTitleGroupGroup

Looking into Leaks…Looking into Leaks…

What is a leak?What is a leak?

Types of LeaksTypes of Leaks

Debugging a Leak Debugging a Leak with Debug with Debug DiagnosticsDiagnostics

Leak Scenarios (2)Leak Scenarios (2)

Why are leaks difficult to identify and Why are leaks difficult to identify and resolve?resolve?

Server applications often cache memory Server applications often cache memory allocationsallocations

The allocations are long termThe allocations are long term

These applications, to outsiders, look to have These applications, to outsiders, look to have memory leaksmemory leaks

There are many other cachesThere are many other cachesUsing Performance Monitor and seeing Using Performance Monitor and seeing memory growth does not equal a memory leakmemory growth does not equal a memory leak

Leak Scenarios (3)Leak Scenarios (3)

What are short-term allocations?What are short-term allocations?Memory expensive applications use “spike” Memory expensive applications use “spike” allocations but are destroyed immediately after allocations but are destroyed immediately after request (not long- sustaining)request (not long- sustaining)

These are These are notnot Memory Leaks Memory Leaks

Leak Scenarios (4)Leak Scenarios (4)

Difference between cache and memory Difference between cache and memory leaks?leaks?

Memory leaks are caused by unbounded Memory leaks are caused by unbounded allocationsallocations

Cache memory usage rises quickly, but tops Cache memory usage rises quickly, but tops out after application is stabilizedout after application is stabilized

Nailing a Memory Leak in a Nailing a Memory Leak in a Web Application using Web Application using DebugDiag: Memory RuleDebugDiag: Memory Rule

NameNameTitleTitleGroupGroup

Session SummarySession SummaryDebugging Live Web Applications without downtime is Debugging Live Web Applications without downtime is challengingchallenging

Crashes cause processes to terminate due to unhandled Crashes cause processes to terminate due to unhandled exceptions exceptions

Hangs are caused due to race conditions, lock contention, Hangs are caused due to race conditions, lock contention, network blocking, database blocking, infinite loops and network blocking, database blocking, infinite loops and deadlocksdeadlocks

Leaks are caused by the failure to release memory Leaks are caused by the failure to release memory allocated beyond typical short term allocations and allocated beyond typical short term allocations and caching caching

Debug Diagnostics has an extensibility model that allows Debug Diagnostics has an extensibility model that allows it to successfully capture and analyze common debugging it to successfully capture and analyze common debugging scenarios to determine the root causescenarios to determine the root cause

Know what DebugDiag can do…and also what it can’t doKnow what DebugDiag can do…and also what it can’t do

Debug Diagnostics 1.0 web Debug Diagnostics 1.0 web resourcesresources

http://www.microsoft.com/http://www.microsoft.com/windowsserver2003/iis/diagnostictools/windowsserver2003/iis/diagnostictools/default.mspxdefault.mspx

© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.