Using CLRMD to replace KD/CDB/NTSD and maybe WinDbg?

May 14, 2015

2 comments

I’ve been using the WinDbg low level debugger for several years now when things get hairy such as looking inside a .NET process and searching for memory leaks and other nasties. Or investigating a dump file, sometimes a kernel crash dump file (created because of the infamous “blue screen of death”), or when faced with a production system where Visual Studio does not (and will not) exist.

Anyone who’s ever used WinDbg (or its equivalent console based debuggers – CDB, NTSD and KD) knows the feeling of wanting to get at some information but not always sure how to do it. What’s the command that can show me this or that? Sure, WinDbg has some GUI, and even some windows that remotely resemble Visual Studio, such as the call stack, processes & threads and local variables, but they’re pretty primitive and in any case don’t give enough information for the typical low level investigation. There are plenty of commands, and the information that is required can probably be obtained with the proper use of these commands, but it’s a steep learning curve, and even after years of usage, some commands are still hidden behind a mountain of command documentation.

I wish there was a nicer version of WinDbg, with better UI that provides more information without the need of executing a lot of commands. I would envision UI written with WPF, so that not only can it be customized easily, but it can be written with clean separation of UI and logic with the MVVM pattern and friends.

The engine that powers all these debuggers is the same – DbgEng.Dll. It packs quite a bit of functionality for attaching to processes, opening dump files and even attaching to a live kernel on another machine; it can set breakpoints, inspect memory, correlate with source code and much more. In fact, the console based debuggers (CDB, NTSD and KD) probably have very little code in them, leveraging DbgEng.Dll for the heavy lifting. This means that anyone can put together a decent console debugger with relatively little amount of work.

The debugger engine exposes a set of COM classes, implementing (no surprise there) a set of COM interfaces – all native of course. So if we wanted to develop the would-be debugger with .NET, we would have to use COM introp wrappers around the interfaces. Unfortunately, the debugger engine does not expose a type library that we can easily import with TlbImp.exe (or simply add a reference with Visual Studio); instead, we must manually create the various interfaces, structs and enums; this is not an easy task – there are quite a few interfaces with many methods.

Fortunately, there is a way to get these interop interfaces without much effort: using CLRMD.

The CLRMD library is a managed library for looking inside .NET processes (and dump files), very similar to using SOS.DLL with one of the aforementioned debuggers, but with a nice, elegant .NET API. CLRMD is interesting onto itself, but this post is not about that functionality in CLRMD. Naturally, we can use CLRMD’s primary functionality as well, giving us access to low level .NET information that can also be displayed nicely in a would-be GUI debugger.

In this post we’ll look at using a side feature of CLRMD – it provides almost complete introp types around the debugger engine interfaces (someone at Microsoft had to do a lot of work!), so these can be used regardless of the primary CLRMD functionality.

So, the first step for creating a WinDbg replacement is already here – the interop debugger engine wrappers, with the added bonus of the primary CLRMD functionality itself.

Creating a dump file analyzer

My final goal for this exploration would be to create a WinDbg replacement, with better UI, but that’s a long road, especially for this post. Let’s start small and see if we can use the debugger engine in a .NET console application to open dump files (either user mode or kernel mode) and use standard debugger commands to analyze it.

The first thing to do is create .NET console application (no fuss here) and add the CLRMD library through NuGet (you must set “include prerelease”) to find the library. CLRMD is actually available on GitHub, and that turns out to be useful, as we’ll see in a moment.

Here’s the view in the NuGet package manager in Visual Studio 2013 after it’s installed:

clip_image001

In the project references, you’ll find the Microsoft.Diagnostics.Runtime assembly, where the entire implementation resides.

Loading the dump file

Now that we have the reference, we can go ahead and load a dump file (let’s assume it’s on the command line) and create a DataTarget derivative:

var target = DataTarget.LoadCrashDump(args[0], CrashDumpReader.DbgEng);
 
var client = target.DebuggerInterface as IDebugClient5;
var control = client as IDebugControl6;

 

Console.Title = “Analyzing “ + args[0];

The later calls get the IDebugClient COM interface for the debugging engine client. The client object also implements a set of IDebugControl interfaces which we’ll need to execute commands entered by the user, as we’ll see in a bit.

The way the debugger client object works is by notifying interested parties when things happen – various debugging events and output from commands. Since we’ll be working with dump files, there is just one event that can occur – an exception if the dump file is a crash dump, which we’ll ignore here. Otherwise, we’re only interested in outputs.

IDebugClient has a SetOutputCallbacks method that accepts an implementation of IDebugOutputCallbacks interface that we must implement. The problem is, that the interop wrappers that CLRMD comes with list the argument as an opaque IntPtr, which is not good enough.

Fixing CLRMD

So I had to clone the Git repository and make the changes so that SetOutputCallbacks (and SetEventCallbacks in case events are of interest) accepts the proper interface instead of the unhelpful IntPtr:

[PreserveSig]
int SetOutputCallbacks([In] IDebugOutputCallbacks Callbacks);

The other methods are similar and had to be replaced in all IDebugClient interface versions, including a “Wide” version that works with Unicode strings, which are always preferred when interoperating with .NET strings.

Now we must remove the NuGet package and add a reference to the modified assembly for CLRMD.

Here’s a simple implementation of the output callback:

class OutputCallbacks : IDebugOutputCallbacksWide {
        public int Output(DEBUG_OUTPUT Mask, string Text) {
               switch(Mask) {
               case DEBUG_OUTPUT.ERROR:
                       Console.ForegroundColor = ConsoleColor.Red;
                       break;
 
               case DEBUG_OUTPUT.EXTENSION_WARNING:
               case DEBUG_OUTPUT.WARNING:
                       Console.ForegroundColor = ConsoleColor.Yellow;
                       break;
 
               case DEBUG_OUTPUT.SYMBOLS:
                       Console.ForegroundColor = ConsoleColor.Cyan;
                       break;
 
               default:
                       Console.ForegroundColor = ConsoleColor.White;
                       break;
               }
 
               Console.Write(Text);
               return 0;
        }
}

We customize the output based on its type and change console color.

Now we need to register the callback object:

client.SetOutputCallbacksWide(new OutputCallbacks());

The Command Loop

Finally, we can create a simple loop to accept commands from the user and execute them:

string input;
do {
        Console.ForegroundColor = ConsoleColor.Green;
        Console.Write("> ");
        input = Console.ReadLine();
 
        control.ExecuteWide(DEBUG_OUTCTL.THIS_CLIENT, input, DEBUG_EXECUTE.DEFAULT);
        
        Console.WriteLine();
} while(input != "q");

Running this with some dump file we can enter commands like k (to show the call stack) or lm (to show the loaded modules):

image

If we enter some unknown command, we get the error in red:

image

One thing that is missing are the extension commands. Trying to enter something like !process shows this:

image

The problem is that the default extensions that are loaded by the standard debuggers are not found in our exe’s path and not in the PATH environment variable. We can use the .load debugger command to load extensions given its full path, or use IDebugControl.AddExtension method in code, prior to invoking commands:

ulong extHandle;
control.AddExtensionWide(@"c:\dbg\x64\winxp\kdexts.dll", 0, out extHandle);

This assumes the particular extension is in the indicated folder We can repeat that for all the standard extension DLLs that come with the Debugging tools for Windows package.

And there you have it. A dump file analyzer with colored output!

For debugging live processes or a live kernel, we need to do a little more work. I’ll leave that to another post.

You can download the code example here.

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*

2 comments

  1. Pingback: Writing a Simple Debugger with DbgEng.Dll | Pavel's Blog

  2. Steven BoneAugust 1, 2017 ב 14:57

    I know this is an old article, and ClrMd has been updated to address the modifications that were required here, but the gist of the article and the one that follows are good. I was running into a problem where symbols were not able to be downloaded with the DbgEng. The root cause turned out to not be obvious at all, so I will post it here.

    Apparently, DataTarget.LoadCrashDump(args[0], CrashDumpReader.DbgEng); will always use the dbghelp.dll, dbgeng.dll, and symsrv.dll that Windows ships by default (system32) which won’t hit symbol servers. The way to work around this is to Pinvoke into LoadLibrary() and provide the full path to the versions of those files (for the bitness of the running application IntPtr.Size is your friend here) shipped with the debugging tools BEFORE you call LoadCrashDump the first time.

    Symbol paths can be set via:

    var symbols = DataTarget.DebuggerInterface as IDebugSymbols;
    //Same as .sympath
    symbols.SetSymbolPath("some symbol path");
    //Same as .exepath
    symbols.SetImagePath("some image path");

    After setting the proper symbol paths you may also need to issue control.ExecuteWide() with these:

    .reload /v /f
    .load sos

    Another hint for loading extensions – I think you could prevent explicit loading of them if you just issue a .extpath followed by a semicolon delimited list of paths to your ‘winext’ and ‘winxp’ directories.

    Prior to disposing DataTarget, make sure to clean up properly:

    IDebugClient5.EndSession(DEBUG_END.ACTIVE_TERMINATE);
    IDebugClient5.SetOutputCallbacks(null);
    IDebugClient5.SetEventCallbacks(null);
    DataTarget.Dispose();

    Reply