DEP, NXCOMPAT, .NET 2.0 SP1
Yesterday I had an interesting case that I thought of sharing, even though there's nothing very new.
In the customer's scenario, a .NET 2.0 GUI application which was functioning perfectly on XP, has stopped working when she moved that application to Vista. Namely, it was getting an access violation in a piece of native code, and terminating unexpectedly. After much grief and looking up some forums on the web, she discovered that a similar problem could be caused by Data Execution Prevention, an important security feature that modern hardware and software work in tandem to implement. (Very briefly, DEP is designed to protect memory pages from being executed, i.e. making it impossible for an application to execute code that is dynamically emitted to a normal data page. This is a security feature because an attacker might emit malicious code into a normal data page and then cause a branch to that page - which is prevented if the page is protected as a no-execute page.)
From this point on, the customer worked hard on disabling DEP for the native ActiveX control that was causing the problem, since it was an ATL control compiled with Visual Studio 6 (old ATL used to generate dynamic thunks into no-execute memory, and then executing them). A much better option would be to recompile the control with Visual Studio 2003/2005/2008 - this would make the problem go away; unfortunately, I repeatedly see organizations that are afraid of porting old, rotting code to the new development tools, sentencing themselves to eternally having a Visual Studio 6 installation on a rusty desktop in some desolate office corner.
Surprisingly, the customer's problems didn't end with disabling DEP for the application, for the simple reason that she wasn't even able to disable DEP for the application. Even when DEP was set up for Opt-In (meaning that only Windows system binaries are covered by DEP, and other applications must explicitly be specified), the exception was still there. Setting DEP to Opt-Out (meaning that applications can opt-out from using DEP) still didn't let us specify that application as an exception. The only workaround was to turn DEP off altogether, which wasn't really an option since the application was a client application, to be installed on workstations with various configurations in an uncontrolled environment (read: people's homes). By the way, we also tried enabling the DisableNX compatibility fix using Application Compatibility Toolkit (check out the KB explanation).
This was starting to seem really fishy, so I asked the following innocent question (seeing that the customer had a very organized build process): "Did you have .NET 2.0 SP1 installed on your build server?"
The answer was a puzzled yes; and then I said, "aha," and we had the solution. Just a couple of days ago, I've read a post documenting a change in the .NET SP1 C# compiler (csc.exe), specifying that it turns on the PE header NXCOMPAT image flag. What's the meaning of that flag, anyway?
Well, if your executable doesn't have the NXCOMPAT flag, it means that it doesn't work with DEP, wasn't tested to work with DEP, or at least you're not 100% sure that you know what kind of relationship you and DEP are going to have. In that scenario, if your application crashes in one of the well-known paths (such as old ATL components), there's an emulation compatibility feature that will get you back up and running.
However, if your executable has the NXCOMPAT flag in its header, it means that you're 100% certain that it works with DEP. Which is a good point of critique towards that barely-documented change in the SP1 C# compiler: if the compiler specifies that default for me, it doesn't mean that I tested my program to work with DEP! 99% of developers and 100% of customers don't even know what DEP or NXCOMPAT is, and they don't know it for a good reason - they shouldn't care. So this has the full potential of a barely-documented breaking change, and I personally feel that the explanation (albeit not an official Microsoft one) is not very convincing:
"Obviously this is not ideal, but aggressively building a computing ecosystem filled with DEP-enabled applications and their accompanying security benefits is very beneficial to Windows users." (from Ed Maurer's post)
So what we had to do is modify the build environment so it doesn't emit the flag. Since there's no way to prevent the compiler from emitting it, you must use the EditBin utility to remove the NXCOMPAT flag from the compiled binary. Since that binary used to be signed, you will need to manually resign it using SN. Both things have to be done as a post-build step; and it clearly stinks.
Just to clarify things a little bit, this has nothing to do with Vista. Coincidentally, the customer started testing the application on Vista simultaneously with deploying .NET 2.0 SP1 to the build server, so she was testing a build with NXCOMPAT. We didn't check, but the exact same thing was supposed to happen on XP. Finally, this also explains why the application couldn't be added to the list of DEP exceptions on the system, or why ACT didn't work - if the executable is marked as NXCOMPAT, it takes precedence over any setting other than turning DEP off altogether.