The Case of the Unit Test and the Native DLL

I got an email from a former client a couple of days ago with roughly the following problem description:

I’m trying to run a unit test (with MSTest) which calls an interop function. The native DLL (written in C) is located in C:\Windows\System32. For some reason, when running the test Visual Studio can’t seem to find the DLL, and throws a FileNotFoundException.

Moreover, I can’t even see the DLL in the File—>Open dialog in Visual Studio! I tried launching Visual Studio as admin and as standard user but the result was the same. What am I missing here?

This was a good opportunity to demonstrate psychic debugging skills. Here’s what I wrote back:

The test probably runs as 32-bit and therefore it looks under C:\Windows\SysWOW64.

Indeed, VSTestHost.exe (the external executable that runs Visual Studio 2008 tests) is compiled with the /32BIT+ CLR header flag, forcing it to run as a 32-bit process even on 64-bit Windows. There are some instructions elsewhere that remove the 32-bit flag from the VSTestHost.exe file, in an obviously unsupported fashion. Finally, Visual Studio 2010 seems to offer a way to run unit tests in a 64-bit process.

Since Visual Studio is a 32-bit process, it is also not a big surprise that the File—>Open dialog “doesn’t work” by not showing the file. Here’s a simple experiment you can perform on your 64-bit Windows:

1. Run two copies of Notepad: C:\Windows\System32\notepad.exe and C:\Windows\SysWOW64\notepad.exe
2. Open an administrative command prompt and copy some text file to C:\Windows\System32
3. Now try opening the file in both Notepad instances using the File—>Open dialog

Bonus reading: More than two years I resolved a similar problem over the phone.

More bonus reading: User-account control redirection can cause similar inconsistencies. A 32-bit application without a manifest that runs without administrative privileges will write a text file to C:\Program Files successfully, but upon closer examination with Windows Explorer you won’t find the file there. Instead, it will be in a redirected VirtualStore folder under %LOCALAPPDATA%. See this great post for more resources, labs, and presentations.