November 2011 - Posts
Another piece of great news delivered at //build/ has to do with device driver development. Coincidentally, a few weeks ago I posted a series of baby-steps with Windows driver development, and if you’ve read some of that you’d notice that the driver dev work is very different from application development – you use a different build environment, you deploy drivers manually, and you debug them with a different debugger.
This story changes, however, with Visual Studio 11. You can now build drivers in Visual Studio, deploy them to test machines automatically, and debug (in kernel-mode!) using the Visual Studio debugger tool windows and source code debugging support.
First things first – you can create driver projects in Visual Studio using the WDK templates. Even if you don’t want the templates, you can create an empty driver, add your files to it, and then compile it:
By the way, if you want to convert an existing driver, there’s a new option File | Convert Sources which will convert your SOURCES file to a Visual Studio solution. I just tried it out with a simple driver, and sure enough Visual Studio created for me a solution, imported my files, and I was able to compile the driver successfully.
Next, you are probably going to debug your driver. Well, you can hit F5 and Visual Studio will ask you how to deploy your driver to a remote machine and how to connect to it for debugging purposes:
For this magic to work, you need several things to snap into place:
- On the target machine, you need to disable UAC completely – using the EnableLUA registry key is the safest way to go, and you need to use a local user account (not a domain or LiveID account).
- On the host machine, you need to run Visual Studio elevated during the configuration process.
- On the host machine, you need to run Visual Studio not elevated when doing the actual debugging.
I hope these are all temporary problems that will be resolved in the final release. At least the elevated-and-then-not-elevated part :-)
But there’s something else you can do even if the automatic deployment doesn’t work out. You can deploy the driver manually and then attach the debugger to the machine. Visual Studio will recognize the symbols and the sources and provide a seamless debugging experience for your kernel-mode code.
So let’s get started. First, in the Tools | Attach to Process dialog we choose the “Windows Kernel Mode Debugger” transport and specify the machine name as a qualifier. In the Find… dialog you can specify the kernel connection details – a simulated serial port in my case connecting to a VMWare virtual machine.
(click the screenshot to enlarge)
After attaching, we can break into the debugger and configure breakpoints and whatnot. The Visual Studio call stack window and source code window are fully functional, but there’s also the Debugger Immediate Window where you can use all the kernel debugger commands from a command line interface. Here’s a screenshot of me stepping through the DriverUnload callback in Visual Studio – note the Locals and Call Stack windows, fully functional:
(click the screenshot to enlarge)
To top it off, Windows 8 features two new debugging transports – USB 3.0 transport, which isn’t all that exciting if it weren’t for the speed and the fact you can use a standard USB cable; and the network transport, which uses the standard network connection to perform kernel debugging. Unfortunately, you need both the host and the target to run Windows 8 for these new transports to work.
This doesn’t even scratch the surface of what beholds driver developers in Windows 8 and Visual Studio 11. Whether you’re doing kernel dump analysis or active driver development, the next Visual Studio is going to help, a lot. I can finally stop explaining in my Windows Internals courses what Visual Studio cannot do, and start showing what it can do.
I have been recently posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn
I had to analyze a memory leak the other day and all I was provided by the customer was a couple of dump files from two points in the application’s lifetime—one dump immediately after initialization, and another dump after the memory leak has accumulated. This is an ideal scenario: now “all it takes” is to compare the two dumps and see which objects are “growing” in memory. That’s, of course, if the leak is actually in managed objects—which isn’t always the case (and VMMap is your friend here).
Naturally, analyzing memory leaks is not always easy, and even if you have a couple of dumps you can spend a very long time in a dark room with WinDbg and SOS and still have no idea where the leak is coming from. Two things that could help immensely with diagnosing memory leaks in complex heap dumps are:
Dominator set analysis—instead of focusing on the myriads of objects occupying memory and trying to figure out the retention chain of each object, dominator set analysis tries to find the objects directly responsible for keeping a large number of other objects alive1. (The Eclipse Memory Analyzer Tool can find dominator sets for large Java heap dumps.)
Visualizing heap dumps—until not long ago, the state of the art in .NET memory dump analysis was running a bunch of WinDbg commands, sometimes assisted by CLR Profiler’s poor visualization features and individual-object-level commands like !vcgroot. Naturally, there are profilers that can visualize snapshots from a live application’s heap; but doing the same for a memory dump is usually impossible.
SciTech’s .NET Memory Profiler can do both. It can import memory dumps as if they were live heap snapshots taken by the profiler, and it can perform dominator set analysis to display the number of objects retained uniquely by another object2.
Full disclosure: I was given a license for the Professional version of the .NET Memory Profiler (v4.0). Nonetheless—and you know this—the following is an unbiased walkthrough through one of the product features.
First things first: we open the Memory Profiler and choose File | Import memory dump. This takes a while—I suspect that the profiler is converting the dump to an internal representation used in the standard snapshots.
We do the same for any additional dumps. Generally, the import process should be linear in the number of objects in the heap dump. When the analysis completes, we can choose the snapshots to compare:
(Naturally, I am not using customer-provided dumps here to prevent information disclosure. Instead, these dumps are taken from a memory leak analysis exercise in the .NET Debugging course.)
Now begins the fun, which I can only describe in a few screenshots and leave the rest to the profiler’s documentation and tutorials. The first thing you’ll see is a list of “issues”—and in this case, I’ve got some instances directly rooted by an EventHandler, and some other instances indirectly rooted by an EventHandler. While this isn’t necessarily a memory leak, it’s certainly suspicious and worth further analysis.
(click the image to enlarge it)
After clicking on MainForm.FileInformation, I get the root paths and an instance list for the suspicious objects. Here’s the root path:
(click the image to enlarge it)
…and here’s the instance table—for each object, the “Held bytes” column is the object’s dominated set’s size:
We see easily that there is a single root path for a large number of instances by clicking the type name. Alternatively, we could focus on the root itself and visualize the graph stemming from it:
To summarize, by allowing to import memory dumps and providing easy visualization, multiple root path traversal, instance- and type-level information, and dominated-set analysis – SciTech Memory Profiler has already helped me pinpoint a real memory leak (in the less than 24 hours since I got the license :-)).
The example above only scratches the surface of what modern profilers can do for you. If you are one of the hardcore pros still using !gcroot and !dumpheap for all your leak analysis needs, that’s great—but if you want to save time and have a tool detect some of the common issues automatically, use a profiler first.
I have been recently posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn
[1] To formalize: for each heap object X there is a (possibly empty) set of roots {R
1, …, R
n} keeping the object alive, i.e. each R
i is a root and there is a reference path from R
i to X. Then, we can define {D
1, …, D
n} the dominator set of X where D
i is the object closest to X on the reference path from R
i to X such that if D
i were removed from the reference path, there would no longer be a path from R
i to X.
[2] I.e., for an object D it can find the dominated set {X1, …, Xm} such that D is in the dominator set of each Xi.
| Yesterday I delivered a 3-hour session on production debugging at the Castra Center in Haifa. The room was set in the middle of a beautiful art gallery, and the projector was playing my slides on a wall surrounded by paintings. Amidst this cozy atmosphere I was talking about some nasty bugs and how to discover them in a production environment. | 
Source: Ezy.co.il
|
Among the topics covered:
Using IntelliTrace in production—we saw an undocumented way to launch IntelliTrace from the command line, capture trace files, and analyze them later in Visual Studio.
Capturing and analyzing dump files—we used Task Manager and Procdump to capture dumps, and then opened them in Visual Studio.
Memory leak detection—we analyzed application memory usage with VMMap and then diagnosed the exact source of a managed memory leak using a memory profiler.
Deadlocks—we approached a hung UI application by capturing a dump and analyzing the deadlock situation with the SOSEX !dlk command. Then, we took a look at another application exhibiting a more complex deadlock with threads and mutexes and used Wait Chain Traversal to resolve it.
If you attended the session, please download the slides and demos and try to walk through some of them on your own. Other resources that may help are linked at my post on debugging resources.
If you haven’t been there, you can always come to my .NET Debugging tutorial day at the upcoming SDP, which will cover these topics and many others—and will be accompanied by hands-on labs to make sure you leave equipped with practical tools for diagnosing your own application errors.
I have been recently posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn
In our C++ Debugging course, there are several scenarios which require WinDbg and cannot be completed in Visual Studio. They all rely on advanced extension commands available in WinDbg. Some examples:
- Tracing opened and closed handles with the !htrace command
- Viewing native heap information with the !heap command
- Loading and executing code in the debuggee process with the SDbgExt extension commands !loaddll, !remotecall
- Inspecting handles to synchronization objects with the !handle command
The sheer power of WinDbg built-in commands and extensions makes it a viable replacement to Visual Studio when doing hard-core debugging. However, the user interface – the tool windows, the source editor, setting up breakpoints, stepping through source, inspecting variables – are no match to the rich Visual Studio environment.
The good news is that as of Visual Studio 11, we’ll be able to use the WinDbg debugging engine (“Windows Debugger”) inside Visual Studio, combining the powerful commands with the convenient interface. Just use the “Windows User Mode Debugger” transport in the Tools | Attach to Process dialog. This also applies to dumps – you can open dumps with the Visual Studio “Minidump Summary” dialog, or you can use the File | Open Crash Dump menu item, the same way as for kernel dumps.
(click the screenshot to enlarge)
That’s really all there’s to it – after attaching, you have the WinDbg shell inside Visual Studio, in the Debugger Immediate Window which we’ve already seen. The Call Stack, Threads, Locals debugger windows are fully functional, and you can combine visual analysis of the source code with running advanced extension commands in the shell.
For example, here’s a thread that is waiting for a mutex:
(click the screenshot to enlarge)
Which thread owns this mutex? The !handle command comes in handy:
(click the screenshot to enlarge)
…and we’ve got the owner information.
Visual Studio 11 is going to change the way we’re doing crash analysis and live debugging of advanced scenarios. I bet plenty of my past and future students are going to be very happy about being able to use Visual Studio instead of learning the ropes of another tool, which has quite the reputation for being user-unfriendly :-)
If you’re doing any .NET debugging, you’re probably wondering if this means better integration for SOS in Visual Studio. Unfortunately, right now the answer is no. With some effort – going through the Visual Studio Tools | Attach to Process dialog – you can attach the WinDbg engine to a .NET process. However, the engine does not recognize .NET any better than standalone WinDbg, so you don’t have managed call stacks, local variables, and source code support within the IDE. But if all you wanted is to run a couple of SOS commands without leaving VS, you certainly can.
Another caveat is that currently you need to install the WDK on top of Visual Studio 11 to get the debugger integration. I’m guessing – or rather, hoping – that in the release version it will suffice to install Debugging Tools for Windows.
I have been recently posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn
Suppose you are sending mass emails (legitimately, no doubt) and want to know which % of recipients actually viewed the email. The standard trick here is to embed a 1x1 image into your email’s HTML source, with the <img src= pointing to a location on your Web server with part of the URL unique to the user (e.g., <img src="http://example.com/track/12345" /> where your mailing system knows that 12345 is associated with john@example.org). When the user opens your email, most email clients will send your server a request for that image*, and voila—you know that the recipient opened it. It’s also quite easy to determine which operating system + mail client combination was used to open the email, by inspecting the HTTP request headers.
But what if you want to track engagement time, i.e. the time the user actually spent reading your email**? This is a valuable metric that can tell you how your messages are working across different groups of users and how engagement times are translated into monetized actions.
This sounds more scary—but here’s another simple trick I’ve seen as recently as yesterday. Embed the same 1x1 image into your HTML source, but this time your Web server needs to be somewhat smarter. When it receives a request for the image, it should return a 302—Moved Temporarily response, with a new URL for the image. When the browser hits that new URL, your Web server returns yet another 302 with yet another URL. This continues until the client gives up—which is usually when the user closes your email.
| Client (browser, email client) | Server |
| GET /track/12345 | 302 Moved Temporarily
Location: …/track/12345?n=1 |
| GET /track/12345?n=1 | 302 Moved Temporarily
Location: …/track/12345?n=2 |
| GET /track/12345?n=2 | 302 Moved Temporarily
Location: …/track/12345?n=3 |
| …you get the idea… | |
The server, on its end, knows the exact times of the requests received from the client and can track not only the fact that the email was opened, but also the engagement time, as defined above.
Is this a legitimate tracking mechanism? In the era of persistent cookies, cookies that stay with you even if you’re logged off the website, tracking that relies on ETags even if you have turned of all cookies and cleared your Flash and HTML5 local storage—it is probably a little naïve to talk about whether a simple image embedded in an email is a legitimate tracking technique, from a morality standpoint.
However, I think it makes sense to discuss the cost of this mechanism, from the client’s perspective. Unless the Web server throttles the request rate by delaying the response as much as possible, the client will issue requests as quickly as it can. Suppose that the RTT is 300ms, and suppose that each request-response exchange takes 1.5KB. That’s a transfer rate of 4.5KB/s, or 270KB per minute. Over a broadband connection at home this is not something to worry about. However, over a GPRS data connection with a limited data plan, or—heavens forbid—a roaming connection, this is a non-negligible amount of data.
Other considerations that come to mind are the CPU and network utilization from a power management perspective. When the email client keeps issuing requests, the network card is probably unable to enter a low power state even if everything else on the system is idle. On mobile devices, tablets, or laptops this may have an effect on battery life (although not as bad as playing Angry Birds :-)).
* Well, actually many email clients have a setting that blocks remote images. And of course there are plain-text email clients that don’t render HTML emails at all. But let’s focus on the users you can track.
** To be accurate, we will see how to determine the time elapsed from the moment the user opened your email to the moment she closed it or switched to another email. Tracking the time the user was actually reading the email requires more sophistication—perhaps eye-movement tracking through Webcam?—that is left as an exercise :-)