Mago DebuggerInner Workings

An implementation overviewBy Aldo Núñez

Mago Debugger

What is debugging? What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger

What is debugging?

Run/Attach Control Inspect Why?

Find out the cause of a problem (bug)

Run/Attach

Kick off a process Attach to an already running process

Control

Breakpoints Stepping Changing instruction pointer Suspend and resume threads

Inspect

Callstack Loaded modules Threads Memory Registers Variables Expressions

Mago Debugger

What is debugging?What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger

What is a debugger?

A process that runs, controls, and inspects another process

Special relationship between debugger and debuggee

System notifies debugger of events taking place in debuggee

OS APIs v. Hardware

OS

• Debug events

• Break/Run mode

• Change state

HW

• Single Step• Breakpoints

• Registers

Break and Run Mode

Break Run

Create or Attach

Break Event

Continue

A Windows Debugger

Loopevent ← WaitForDebugEvent( timeout )if got event

ContinueDebugEvent( event.pid,event.tid,

DISCARD_EXCEPTION )Until event.code = EXIT_PROCESS

Debug Events

Start Process Exit Process Start Thread Exit Thread Load Module Unload Module Exception Message

Windows API for Debugging

Debug Events• WaitForDebugEvent• ContinueDebugEvent

Launch• CreateProcess• TerminateProcess

Attach• DebugActiveProcess• DebugActiveProcessS

top

Registers• GetThreadContext• SetThreadContext

Memory• ReadProcessMemory• WriteProcessMemory

Threads• SuspendThread• ResumeThread

OS and HW Co-op (x86): Single Step

Enable SS

• GetThreadContext• context.Eflags or

0x100• SetThreadContext• ContinueDebugEvent

SS Event

• event ← WaitForEvent• event.code =

EXCEPTION• event.exception.code

= EXCEPTION_SINGLE_STEP

OS and HW Co-op (x86): Breakpoint

Enable BP

• ReadProcessMemory• WriteProcessMemory

• data = 0xCC (int 3)• FlushInstructionCache• ContinueDebugEvent

BP Event

• event ← WaitForEvent• event.code =

EXCEPTION• event.exception.code

= EXCEPTION_BREAKPOINT

• Except. address = original

• EIP = original + 1

Mago Debugger

What is debugging? What is a debugger?What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger

What is Mago?

A debugger for D programs A set of independent libraries A Visual Studio plug-in

History

Interest in debuggers since 2005 Started September 2009 Source code released August 2010 Integrated into Visual D September

2010

Libraries v. Visual Studio plug-in

Benefits to making separate components Targeted testing Mix and match for different purposes Use with any shell program

Benefits to making VS plug-in Well tested shell program already

written High level debug programming model

What does Mago look like?

Debug Engine

Exec Expr Syms

Component Responsibilities

Exec•Control debuggee•Read and change state

Expr•Evaluate D Expressions•Formatting

Syms•Read debug info

DE•Combine other components•Expose AD7 interface

Mago Debugger

What is debugging? What is a debugger? What is Mago?Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger

Execution Agent

Abstracts run, control, and inspection services

Built first to make it as solid as possible

Many APIs are locked to thread that started debuggee Because of underlying Windows API

Services

WaitForEvent, Continue from event Launch, Terminate Attach, Detach Read, Write Memory Set, Remove Breakpoint Step, Cancel Step Async Break

Breakpoint Management

Software breakpoint abstraction Hardware breakpoint abstraction Breakpoint sharing Resuming from breakpoint

Breakpoint Lifecycle

inc

mov

inc

mov

inc

mov

incinc

inc

Patch Run/Hit BP

TemporaryUnpatch

Single Step/Restore BP

Unpatch

IP

IP

IP

int

int

int

Multithreaded Single-Step

Stepping over a single instruction Can easily step over most

instructions with native single step (SS)

Others require setting a BP after the instruction REP string instructions

Single-step with SS

A

B

A

B

Q

R

SS

SS event

SS TargetThread

Hardware is set up to single step in this thread only.When that SS is over, hardware fires SS event for this thread only.

Single-step with BP: Bad

A

B

A

B

Q

R

run

BP event

SS TargetThread

At hardware level, BPs are scoped to memory, not threads.So, any thread can hit the BP.

Single-step with BP: Good

A

B

A

B

Q

R

run

BP event

SS TargetThread

Force scoping BP to target thread by suspending all others.

Steppers

State machines for complex stepping In, Over, Out, Go/Resume Instruction, Statement

Control low-level SS and BP Receive notification of SS and BP

events Can be canceled

Stepping Scenarios

Instruction steppers handle 18 scenarios 3x Instruction type: (simple, call, REP) 2x At a BP 3x Movement: (Go, Step In, Step Over)

Range stepper uses instruction steppers over an address range

Step Out stepper runs to a BP at return address

Example: Call Instruction, At BP

Go/Resume

IP

Step In

IP

Step Over

IP

IP

IP SS=1

SS=1

SS

SS

IP SS=1

BP

BP

BP

XA

XA

XA

BP

BP

BP

SS

BP

BP

BPBP BP BP

IPXA

BPBP

BPIP

BP

Threading

Debugger Thread

Main Thread

Run Command

Dispatch Event

Command

EventsDebuggee

Mago Debugger

What is debugging? What is a debugger? What is Mago? Execution AgentExpression Evaluator Symbol Reader Debug Engine D and the debugger

Expression Evaluator

Evaluates D expressions Input is textual expression Output is a result value record Declarations, symbols, and input

values come from outside IValueBinder, IDeclaration

Handles formatting values Enumerates children of values Based on DMD front end

EE Data Flow

chars tokensScanner Parser Value Binder

Process Debug Info

Value

Node Tree

EE Usage

MakeTypeEnv( &typeEnv );MakeNameTable( &nameTable );ParseText( L”a[2] + 3”, typeEnv,

nameTable, &expr );expr->Bind( options, binder );expr->Evaluate( options, binder, &result );

Resulting Node Tree

Parser

Type Env

String Table

Binder

Add

Int 2ID "a"

Index Int 3

Find,Make Type

Add

Parse "a[2] + 3"

Make

Make

Build

debuginfoFind,

Make Type

Find ObjectFind Symbol

Debuggee Memory

9 3 5 1

1A334

Read

Variable "a"Array of intat 1A334

Get value at addr 1A33C

EvalBind Value = 8

Mago Debugger

What is debugging? What is a debugger? What is Mago? Execution Agent Expression EvaluatorSymbol Reader Debug Engine D and the debugger

Symbol Reader

Reads debug info for a program Maps of source files to lines Maps of source code lines to addresses Functions – address and scopes Symbols – name, type, value, storage Types

Reads specific formats Currently, CodeView 4.10, output by

DMD

Compare to DWARF

CODEVIEW

Fixed record fields Numeric constant

compression Common type

encoding Sorted symbols Nested Lexical blocks

DWARF

Flexible Attributes: key-value Explicit base type

definition Location expressions

Compression Flatten tree Abbreviations Byte code for tables

CodeView Sample

12

S_GDATA32

0x00000310

1

0x1003

a

IntArray1

LF_ARRAY

T_INT4 (0x0074)

T_UINT4 (0x0075)

0x0010

18

record length

record type

offset

segment

@type

name

name

record type

@elemtype

@idxtype

byte count

record length2

2

2

*

*

2

2

4

2

2

*

2

Spec Actual

1

9

Size

Mago Debugger

What is debugging? What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol ReaderDebug Engine D and the debugger

Debug Engine

A plug-in to the VS Debugger package (vsdebug.dll)

Standalone DLL doesn’t depend on any other package

Expected to implement AD7 interface Knows how to debug one kind of

program DEs are multiplexed during a debug

session

AD7 Interface

A programming model for debugging processes

Single-threaded calls from VS Debugger to DE Simplifies design

COM interfaces Debug Engine is a COM co-class

Programming Model

IDebugEngine2 IDebugThread2 IDebugBoundBreakpoint2 IDebugExpression2 IDebugStackFrame2 IDebugDisassemblyStream2 IDebugEvent2

Threading

Event

HandleEvent

Event

HandleEvent

Main

SDM DE Event Monitor AD7 Callback

Mago Debugger

What is debugging? What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug EngineD and the debugger

D and the debugger

Rewrite in D eventually Only EE and parts of DE know about

D EE Test input generated by D

program Expression and expected value Uses compile-time reflection

GenUnaryTest.dvoid main( string[] args ) {

writeln( "<test>" );

if ( set == 1 )

UnaryList!(byte, ubyte, short, ushort, int, uint, long, ulong).Operation( op );

else if ( set == 2 )

UnaryList!(float, double, real, ifloat, idouble, ireal, cfloat, cdouble, creal).Operation( op );

writeln( "</test>" );

}

template Unary(T) {

void Unary( void function( T ) func ) {

foreach ( t; Vals!T.vals ) {

Id++;

writefln( " <verify id=\"%s_%d\">", Prefix, Id );

func( t );

writefln( " </verify>" );

}

}

}

template UnaryList(T...) {

void Operation( Op op ) {

foreach ( t; T ) {

switch ( op ) {

case Op.Negate: Unary!(t)(&UnOp!(t).Negate); break;

case Op.BitNot:

static if (__traits( compiles, Unary!(t)( &UnOp!(t).BitNot )))

Unary!(t)( &UnOp!(t).BitNot );

break;

}

}

}

}

template Vals(T) {

static T[] vals;

static this() {

static if ( !__traits( isFloating, T ) )

// Add values like T.max, cast(T) -1, cast(T) 0

else

// Add values like T.nan, -T.infinity, cast(T) 0

}

}

GenUnaryTest.dtemplate UnOp(T) {

static if ( __traits( isIntegral, T ) )

void BitNot( T t ) {

writeln( " <bitnot>" );

CastTerm( t );

writeln( " </bitnot>" );

writeln( " <typedvalue>" );

PrintType!(typeof( ~t ))();

PrintTerm( ~t );

writeln( " </typedvalue>" );

}

void PrintType(X)() {

static if ( __traits( isArithmetic, X ) )

writefln( " <basictype name=\"%s\"/>", typeid( X ) );

else

writefln( " <reftype name=\"%s\"/>", typeid( X ) );

}

void CastTerm(X)( X x ) {

writeln( " <cast>" );

PrintType!X();

PrintTerm( x );

writeln( " </cast>" );

}

void PrintTerm(X)( X x ) {

static if ( is(X==creal) || is(X==cdouble) ||

is(X==cfloat) ) {

writeln( " <group>" );

writeln( " <add>" );

writefln( " <realvalue value=\"%a\"/>", x.re );

writefln( " <realvalue value=\"%ai\"/>", x.im );

writeln( " </add>" );

writeln( " </group>" );

}

else if ( is( X == ireal ) || is( X == idouble ) || is( X == ifloat ) ) {

writefln( " <realvalue value=\"%ai\"/>", x );

}

else if ( __traits( isFloating, X ) ) {

writefln( " <realvalue value=\"%a\"/>", x );

}

else if ( is( X == ulong ) || (is( X == long ) && (x < 0)) ) {

writefln( " <intvalue value=\"%dUL\"/>", x );

}

else

writefln( " <intvalue value=\"%d\"/>", x );

}

}

Implementation Challenges

Interface for sharing between modules

Take DMD front end or mimic it 80-bit floating point in expression

eval Disassembling variable length

instructions

Links

Mago: http://dsource.org/projects/mago_debugger

D: http://digitalmars.com/d/ Visual D IDE by Rainer Schuetze:

http://dsource.org/projects/visuald Visual Studio Debugger Extensibility:

http://msdn.microsoft.com/en-us/library/bb161718.aspx

x86 and x64 Manuals: http://www.intel.com/products/processor/manuals/

DWARF Format: http://www.dwarfstd.org