Writing a Simple Debugger with DbgEng.Dll

July 27, 2015

no comments

In my post on using CLRMD’s debugger engine wrappers to “debug” a dump file, I’ve shown how we can take advantage of the documented API of DbgEng.Dll – the debugger engine that drives the Microsoft debuggers – CDB, NTSD, KD and WinDbg. In this post, we’ll take a step further and create a basic functioning user mode debugger that is able to attach to a process and do “normal” debugging, somewhat similar to CDB/NTSD but with some small colorful bonuses.

As you may recall, I’ve taken the CLRMD project and made some enhancements to the callback interop types defined there to allow registering for event callbacks and output callbacks. Also, I made the DebugCreate function public so that we can create the debugger client object without going through CLRMD, because in the general case, when the process to debug may or may not be .NET based, it seems more logical to go through the general case; it’s still possible to get the CLRMD DataTarget object from the debugger client’s interface if so desired.

Our debugger will be a console application (similar to CDB/NTSD), but we’ll add some colors to the output to make things a bit more interesting (and colorful).

We’ll start by creating a Console application named SimpleDebugger. We’ll add a reference to the enhanced CLRMD assembly. We’ll expect a process ID to attach to at the command line or an executable path to launch and attach to.

But first we need to create the debugger client object and extract the IDebugClient5 and IDebugControl6 interface (these are the most derived of their kind, although for our purposes one of the less-derived interfaces will do):

Guid guid = typeof(IDebugClient).GUID;

 

// create debug client object

object obj;

CheckHr(NativeMethods.DebugCreate(ref guid, out obj));

var client = obj as IDebugClient5;

var control = client as IDebugControl6;

 

CheckHr is a simple method that throws an exception if the passed HRESULT is negative, indicating some failure. This should not happen in this case if the debugging engine DLL exists.

One thing to note is that the Main method is not decorated with the [STAThread] attribute (as it is in a WinForms/WPF application), which means the Debug Client object is created in the MTA (in COM terms), as it should. In general, creating the Debug Client in the MTA is almost mandatory, otherwise in a GUI app deadlocks may happen in accessed from the UI thread, and exceptions will be thrown if accessed from another STA or the MTA; we’ll keep simple here and work with the MTA only, which is fine for a Console Application).

Next, we need to register for events coming from the engine and for outputs that we should echo to the console:

// register event and output callbacks

 

var events = new EventCallbacks(control);

client.SetEventCallbacksWide(events);

client.SetOutputCallbacksWide(new OutputCallbacks());

The EventCallbacks class implements the IDebugCallbackEvents (or IDebugCallbackEventsWide) interface, which we’ll look at a bit later. The OutputCallbacks class implements the IDebugOutputCallbacks(Wide) interface. Its implementation is fairly simple:

class OutputCallbacks : IDebugOutputCallbacksWide {

      public int Output(DEBUG_OUTPUT Mask, string Text) {

            switch(Mask) {

            case DEBUG_OUTPUT.DEBUGGEE:

                  Console.ForegroundColor = ConsoleColor.Gray;

                  break;

 

            case DEBUG_OUTPUT.PROMPT:

                  Console.ForegroundColor = ConsoleColor.Magenta;

                  break;

 

            case DEBUG_OUTPUT.ERROR:

                  Console.ForegroundColor = ConsoleColor.Red;

                  break;

 

            case DEBUG_OUTPUT.EXTENSION_WARNING:

            case DEBUG_OUTPUT.WARNING:

                  Console.ForegroundColor = ConsoleColor.Yellow;

                  break;

 

            case DEBUG_OUTPUT.SYMBOLS:

                  Console.ForegroundColor = ConsoleColor.Cyan;

                  break;

 

            default:

                  Console.ForegroundColor = ConsoleColor.White;

                  break;

            }

 

            Console.Write(Text);

            return 0;

      }

}

The interface has just one method – Output. It receives a mask, indicating the kind of output and the output itself. For fun, I’ve changed the foreground color based on the output type (mask). Here’s a screenshot showing some of these colors in action:

Some Colors while Debugging

Now that the callbacks are registered, we can move on to attaching to a running process.

One thing before that – when the target process is running – how can we break into the target without previously setting a breakpoint? We’ll use the console Ctrl+C key sequence to break into the debugger:

// register for Ctrl+C to break into the debugger

 

Console.CancelKeyPress += (s, e) => {

      e.Cancel = true;

      control.SetInterrupt(DEBUG_INTERRUPT.ACTIVE);

};

The IDebugControl.SetInterrupt method attempts to break into the target asynchronously. After the call, we expect to get an event indicating a breakpoint was hit (as we’ll see later).

Now we’re ready to read the command line arg and attach to or create and attach to the process:

// read command line arg (PID or EXE) and start debugging

 

uint pid;

if(uint.TryParse(args[0], out pid)) {

      // start debugging by attaching to a process

 

      CheckHr(client.AttachProcess(0, pid, DEBUG_ATTACH.DEFAULT));

}

else {

      // start debugging by creating and attaching to a process

 

      CheckHr(client.CreateProcessAndAttachWide(0, args[0],

            (DEBUG_CREATE_PROCESS)DEBUG_PROCESS.ONLY_THIS_PROCESS,

0, DEBUG_ATTACH.DEFAULT));

}

To do the actual attach, the IDebugControl.WaitForEvent must be called:

// wait for the initial "attach" event

 

CheckHr(control.WaitForEvent(DEBUG_WAIT.DEFAULT, uint.MaxValue));

The debugger engine is essentially a state machine. It maintains an execution state and other data related to the attached processes. What we need is a loop that works as long as there is a target. This is our loop:

DEBUG_STATUS status;

int hr;

 

while(true) {

      CheckHr(control.GetExecutionStatus(out status));

      if(status == DEBUG_STATUS.NO_DEBUGGEE) {

            Console.WriteLine("No Target");

            break;

      }

 

      if(status == DEBUG_STATUS.GO || status == DEBUG_STATUS.STEP_BRANCH ||

status == DEBUG_STATUS.STEP_INTO ||

status == DEBUG_STATUS.STEP_OVER) {

            hr = control.WaitForEvent(DEBUG_WAIT.DEFAULT, uint.MaxValue);

            continue;

      }

 

      if(events.StateChanged) {

            Console.WriteLine();

            events.StateChanged = false;

            if(events.BreakpointHit) {

                  control.OutputCurrentState(DEBUG_OUTCTL.THIS_CLIENT,

DEBUG_CURRENT.DEFAULT);

                  events.BreakpointHit = false;

            }

      }

 

      control.OutputPromptWide(DEBUG_OUTCTL.THIS_CLIENT, null);

      Console.Write(" ");

      Console.ForegroundColor = ConsoleColor.Gray;

      string command = Console.ReadLine();

      control.ExecuteWide(DEBUG_OUTCTL.THIS_CLIENT, command,

DEBUG_EXECUTE.DEFAULT);

}

I must admit that this loop is not at all obvious, I’m not yet fully convinced it works perfectly; at least it does with the testing I did. The basic premise is this: if the target is running in some way (normal run, step over, step into, etc.) – wait for a debugger event (most likely some breakpoint, but other things possible such as the target process existing or terminating in some way.

Otherwise, we are at a breakpoint for some reason: show some info to the user if the breakpoint was just hit (OutputCurrentState), and then display the prompt (OutputPromptWide – this calls back to the output callbacks (possibly) asynchronously) and then get a command from the user and execute it (ExecuteWide). Repeat until there is no target anymore (GetExecutionStatus returns DEBUG_STATUS.NO_DEBUGGEE).

The missing piece of the puzzle is the event callbacks that happen during the run of this loop, called by the engine at appropriate times. This is how the StateChanged and BreakpointHit fields above change values.

The IDebugEventCallbacks(Wide) is a much more interesting interface than IDebugOutputCallbacks. The first method called by the engine is GetInterestMask, where the object indicates what kind of events are of interest. For this example, I’ve requested all of them, but ignored most of them:

class EventCallbacks : IDebugEventCallbacksWide {

      readonly IDebugControl6 _control;

 

      public EventCallbacks(IDebugControl6 control) {

            _control = control;

      }

 

      public int GetInterestMask(out DEBUG_EVENT Mask) {

            Mask = DEBUG_EVENT.BREAKPOINT | DEBUG_EVENT.CHANGE_DEBUGGEE_STATE

| DEBUG_EVENT.CHANGE_ENGINE_STATE | DEBUG_EVENT.CHANGE_SYMBOL_STATE |

DEBUG_EVENT.CREATE_PROCESS | DEBUG_EVENT.CREATE_THREAD | DEBUG_EVENT.EXCEPTION | DEBUG_EVENT.EXIT_PROCESS |

DEBUG_EVENT.EXIT_THREAD | DEBUG_EVENT.LOAD_MODULE |

                  DEBUG_EVENT.SESSION_STATUS | DEBUG_EVENT.SYSTEM_ERROR |

                  DEBUG_EVENT.UNLOAD_MODULE;

 

            return 0;

      }

You can get a sense of the possible events just by looking at the DEBUG_EVENT enum values.

Here are the implementations of Breakpoint and Exception methods where they indicate back to the engine what’s its new requested state:

public int Breakpoint(IDebugBreakpoint2 Bp) {

      BreakpointHit = true;

      StateChanged = true;

      return (int)DEBUG_STATUS.BREAK;

}

 

public int Exception(ref EXCEPTION_RECORD64 Exception, uint FirstChance) {

      BreakpointHit = true;

      return (int)DEBUG_STATUS.BREAK;

}

Notice that in the Exception case, we get indication if this is a first chance or second chance exception. Currently, we break on both. In a real debugger, we would probably let first exceptions continue execution (unless otherwise specified by the user) because they may be handled normally. Another caveat to watch for is that Step Over and Step Into are exceptions as well; I’ll leave that to the interested reader to handle correctly (hint: the exception type is provided as well).

When a debugger attaches to a process, the CreateProcess event method is called. We can ignore the event, but we can also set a breakpoint to stop at that initial attach:

public int CreateProcess(ulong ImageFileHandle, ulong Handle,

ulong BaseOffset, uint ModuleSize, string ModuleName, string ImageName,

      uint CheckSum, uint TimeDateStamp, ulong InitialThreadHandle,

ulong ThreadDataOffset, ulong StartOffset) {

 

      IDebugBreakpoint2 bp;

      _control.AddBreakpoint2(DEBUG_BREAKPOINT_TYPE.CODE,

uint.MaxValue, out bp);

      bp.SetOffset(StartOffset);

      bp.SetFlags(DEBUG_BREAKPOINT_FLAG.ENABLED);

      bp.SetCommandWide(".echo Stopping on process attach");

 

      return (int)DEBUG_STATUS.NO_CHANGE;

}

In a full-fledged debugger we would probably allow the user to indicate if such a breakpoint is desired.

The rest of the events methods are not that interesting for our purposes, but would be interesting in a more complete debugger implementation.

So there you have it. A debugger, with breakpoints and all:

Debugger Session with a Breakpoint

To make this work, we must copy the required DLLs from the debugging tools for windows folder (with the correct “bitness”) to the output of our debugger: dbgeng.dll, dbghelp.dll, srcsrv.dll, symsrv.dll. The latter is needed for communicating with symbol servers, such as the Microsoft Symbol Server. Also, if we want access to the default extension commands (e.g. !handle), we need to copy the winext and winxp folders from the tools to the SimpleDebugger.Exe’s folder.

The next logical step would be to create a full fledged debugger with a proper GUI in WPF that provide better UI and many graphic windows for various debugger-related information such as threads, stacks, handles, memory, locks and much more without the need to know every little details of every command as is sometimes necessary when working with the existing debuggers.

I’ve actually started such a project… we’ll see how it goes. For now, you can download the source and binaries for this sample (only the 64 bit parts were built and copied).

SimpleDebugger Sample

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*