.NET Debugging for the Production EnvironmentPart 4: Common Debugging Commands

Brad Linscott

Premier Field Engineeringing

AgendaSymbols

Commands

Common debugging commands: miscellaneous

Common debugging commands for hangs

Common debugging commands for exceptions

Common debugging commands for memory pressure

3

Not needed to resolve managed call stacksNeeded to resolve native call stacks

Translation: Not always mandatory, but in general, a good idea to ensure symbols are lined up for managed issues

File, Symbol File Path; .sympath; .symfixhttp://msdl.microsoft.com/download/symbols

Symbols

4

CommandsMeta commands (.)Extension commands(!)

Breakdown of Debugger Commands

5

| - pipe command displays command line of process0:029> |

. 0 id: 5ec examine name: c:\WINDOWS\system32\inetsrv\w3wp.exe

vertarget shows dump time, OS version, process’ lifetime, and more0:029> vertarget

Windows Server 2003 Version 3790 (Service Pack 2) MP (2 procs) Free x86 compatible

Product: Server, suite: TerminalServer SingleUserTS

kernel32.dll version: 5.2.3790.4062 (srv03_sp2_gdr.070417-0203)

Machine Name:

Debug session time: Tue Jul 29 10:51:49.000 2008 (GMT-5)

System Uptime: 0 days 6:19:45.750

Process Uptime: 0 days 4:34:57.000

Kernel time: 0 days 0:24:43.000

User time: 0 days 0:05:48.000

Common Miscellaneous commands

6

.load <extension> loads a debugging extension

.unload <extension> unloads a debugging extension.chain shows loaded extensions, most recently-loaded is listed first.0:000> .chain

Extension DLL search Path:

C:\debuggersx86\WINXP;C:\debuggersx86\winext;C:\debuggersx86\winext\arcade

psscor2.dll: image 2.0.50826.0, API 1.0.0, built Wed Oct 29 12:48:12 2008

[path: C:\debuggersx86\psscor2.dll]

dbghelp: image 6.13.0009.1140, API 6.2.6, built Tue May 17 18:15:43 2011

[path: C:\debuggersx86\dbghelp.dll]

ext: image 6.13.0009.1140, API 1.0.0, built Tue May 17 18:16:08 2011

[path: C:\debuggersx86\winext\ext.dll]

exts: image 6.13.0009.1140, API 1.0.0, built Tue May 17 18:16:36 2011

[path: C:\debuggersx86\WINXP\exts.dll]

uext: image 6.13.0009.1140, API 1.0.0, built Tue May 17 18:16:03 2011

[path: C:\debuggersx86\winext\uext.dll]

Common Miscellaneous commands (cont’d)

7

!help displays help for top-most extension!psscorX.help <command> displays detailed help for <command>

.hh <command> opens debugger.chm to the <command> topic. <command> can’t be from an extension outside the Debugging Windows for Tools package!peb shows loaded modules, environment variables, command line arg, and more

Common Miscellaneous commands (cont’d)

8

!runaway shows quanta of each thread0:029> !runaway

User Mode Time

Thread Time

17:828 0 days 0:04:40.578

11:380 0 days 0:04:24.046

14:288 0 days 0:04:14.296

13:4a0 0 days 0:03:58.984

29:13e4 0 days 0:01:13.078

~ displays PID and TID; ~#s switches the active thread to thread #0:000> ~

. 0 Id: 2c14.2048 Suspend: 0 Teb: 7ffde000 Unfrozen

1 Id: 2c14.35b8 Suspend: 0 Teb: 7ffdd000 Unfrozen

2 Id: 2c14.32f8 Suspend: 0 Teb: 7ffdc000 Unfrozen

3 Id: 2c14.3754 Suspend: 0 Teb: 7ffd9000 Unfrozen

Red – logical Thread ID Blue – Hex Thread ID

Purple – Process ID Green – beginning address of Thread Environment Block

Common Miscellaneous commands (cont’d)

9

k, kn, kb, kP, etc displays the native stack in various ways0:006> kbnL

# ChildEBP RetAddr Args to Child

00 0114fcec 7c827b89 7c83c5fe 0000001c 0114fd34 ntdll!KiFastSystemCallRet

01 0114fcf0 7c83c5fe 0000001c 0114fd34 00000001 ntdll!ZwWaitForMultipleObjects+0xc

02 0114ffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWaitThread+0x161

03 0114ffec 00000000 7c83c4b3 00000000 00000000 kernel32!BaseThreadStart+0x34

!clrstack is from psscorX; displays managed call stack0:055> !clrstack

OS Thread Id: 0x1ea0 (55)

ESP EIP

386fe4a4 7c82847c [HelperMethodFrame_1OBJ: 386fe4a4] System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle[], Int32, Boolean, Boolean)

386fe570 7928b3ff System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)

386fe590 129dd8cb System.Net.TimerThread.ThreadProc()

386fe5dc 792d6e46 System.Threading.ThreadHelper.ThreadStart_Context(System.Object)

386fe5e8 792f5781 System.Threading.ExecutionContext.runTryCode(System.Object)

<clip>

Common Miscellaneous commands (cont’d)

10

~*k; ~*e!clrstack displays all the stacks

!syncblk finds outstanding syncblocks on which other threads are waiting||0:0:000> !syncblk

Index SyncBlock MonitorHeld Recursion Thread ThreadID Object Waiting

3724 0x06c7b13c 59 1 0x101dc0 0x120c 14 0x1c2643f8 System.String

Waiting threads: 22 23 24 25 27 30 31 39 40 45 50 51 52 53 54 55 56 57 59 60 61 62 63 64 65 66 67 68 70

3752 0x02b382a0 9 1 0x6d5a4b8 0x1390 70 0x140159ec System.String

Waiting threads: 69 74 78 79

-----------------------------

Total 3942

ComCallWrapper 7

ComPlusWrapper 3

ComClassFactory 0

Free 234

Common commands for hangs

11

!finalizequeue from psscorX lists objects registered for finalization.• 0:000> !finalizequeue

SyncBlocks to be cleaned up: 524MTA Interfaces to be released: 0STA Interfaces to be released: 0----------------------------------generation 0 has 577 finalizable objects (0e480940->0e481244)generation 1 has 41 finalizable objects (0e48089c->0e480940)generation 2 has 543 finalizable objects (0e480020->0e48089c)Ready for finalization 151459 objects (0e481244->0e5150d0) A number > 0 indicates possible problems. A number this big indicates a definite problem.Statistics:<clip>

• If this is the case, investigate the Finalizer thread (!threads is one way to find it) to determine cause of blockage.

Common commands for hangs, cont’d

12

!dae from psscorX displays all managed exceptions in the GC heaps

Double-check single OutOfMemoryException, StackOverflowException, and ExecutionEngineException instances

!pe <exception addr> from psscorX provides more detailed information on a specific managed exception.

Common commands for exceptions

13

To a large extent, debugging crash dumps of a managed & native app don’t differ by much Upon opening dump, active thread is faulting thread. .!u for deeper analysis of the disassemblyNative debugging skills as well commonly needed to determine root cause

Common commands for exceptions, cont’d

14

!address shows information about a particular address (type, protection, state, usage)0:000> !address -summary-------------------- Usage SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Pct(Busy) Usage 2ec53000 ( 766284) : 36.54% 72.26% : RegionUsageIsVAD 3f451000 ( 1036612) : 49.43% 00.00% : RegionUsageFree 83fb000 ( 135148) : 06.44% 12.74% : RegionUsageImage<clip> Tot: 7fff0000 (2097088 KB) Busy: 40b9f000 (1060476 KB)-------------------- Type SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Usage 3f451000 ( 1036612) : 49.43% : <free> 8cb5000 ( 144084) : 06.87% : MEM_IMAGE 1aa64000 ( 436624) : 20.82% : MEM_MAPPED 1d486000 ( 479768) : 22.88% : MEM_PRIVATE-------------------- State SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Usage 32628000 ( 825504) : 39.36% : MEM_COMMIT 3f451000 ( 1036612) : 49.43% : MEM_FREE e577000 ( 234972) : 11.20% : MEM_RESERVE

Largest free region: Base 6e0e0000 - Size 03970000 (58816 KB)

1GB free and your largest contiguous FREE is 58MB is a good sign of VA fragmentation

Common commands for memory pressure

15

A managed leak/bloat is usually in one of 2 places:

GC Heaps (!eeheap –gc)Loader heaps (!eeheap –loader)

Common commands for memory pressure

16

Loader heaps:Contain type definitions and other internal structures~20MB is usually considered high for a busy w3wpIf !dda results in too many assemblies (generally, more than a few hundred means this is contributor to high mem), find why they’re being created!dumpdomain lists assemblies loaded in each appdomainIn w3wp, !finddebugtrue tells you if <compilation debug=“true”/>, which is a root cause of high memoryQ886385, Q316775 are common root causes

Common commands for memory pressure, cont’d

17

!eeheap –gcBreaks down mem usage among Gens within GC heaps. Helps you to see if memory issue is concentrated in a specific generation or the LOH.!dumpheap –stat may help. Focus on objects with high Count and/or high Total Memory. !gcroot <mt address> may help find root cause!clrusage may also give helpful insight

||0:0:000> !help clrusage-------------------------------------------------------------------------------!CLRUsage [-v]

!CLRUsage dumps out the memory that .NET is using. This is different from !EEHeap -gc in that it gives how much memory is committed and reserved andalso gives the initial allocation that we made.

Common commands for memory pressure, cont’d

18

Many commands are used regularly, this session has covered just a fewBe curious – use !help frequentlyPractice, Practice, Practice

Conclusion