P/Invoke with C++ bool Return Values

January 17, 2013

tags: ,

I encountered an interesting gotcha today, which I thought would be interesting to share with you. The symptoms were the following: I wrote a simple C++ function exported from a DLL, and tried to invoke it using a P/Invoke wrapper. The C++ function returns a bool, and was declared as follows (the code was simplified for expository purposes):

extern "C" __declspec(dllexport) bool IsPrime(int n) {
    if (n <= 1) return false;
    if (n == 2) return true;
    if (n % 2 == 0) return false;
    for (int i = 3; i < n; ++i) {
        if (n % i == 0) return false;
    return true;

The managed counterpart was:

static extern bool IsPrime(int n);

At runtime, for some values of n, the output would be inconsistent. Specifically, when n=0 or n=1, the managed wrapper would return true – although the C++ function clearly returns false. For other values, however, the wrapper returned the right values.

Next, I disassembled the C++ function, to find the following code responsible for the first branch:

cmp dword ptr[n], 1
jg BranchNotTaken
xor al, al
jmp BailOut



In other words, to return false, the function clears out the AL register (which is the lowest byte of the EAX register), and then returns. What of the rest of the EAX register? In the debug build, it’s likely to contain 0xCCCCCCCC, because of this part of the function’s prologue:

lea edi,[ebp-0CCh] 
mov ecx,33h 
mov eax,0CCCCCCCCh 
rep stos dword ptr es:[edi]

To conclude, the function returns 0xCCCCCC00 in the EAX register, which is then interpreted as true by the managed wrapper. Why? Because by default, P/Invoke considers bool parameters to be four-byte values, akin to the Win32 BOOL, and then any non-zero value is mapped to true.

To fix this, I had to convince P/Invoke to marshal back the return value as a single byte. There’s no dedicated type for that in the UnmanagedType enumeration, but U1 (unsigned byte) or I1 (signed byte) will do the trick:

[return: MarshalAs(UnmanagedType.U1)]
static extern bool IsPrime(int n);

I am posting short links and updates on Twitter as well as on this blog. You can follow me: @goldshtn

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>



  1. GlenJanuary 17, 2013 ב 1:31 PM

    Been caught out by that one. btw FXCop has a rule to catch them – CA1414
    “UnmanagedType.U1 should be used for C++ bool or other 1-byte types.”

  2. Daniel RoseJanuary 18, 2013 ב 7:16 AM

    That is why there is an FxCop warning, CA1414, if you don’t annotate bool arguments with MarshalAs. See http://msdn.microsoft.com/en-us/library/ms182206.aspx

    I found similar errors in C code when a short was passed to a function needing an int (not caught because “we don’t need prototypes”).

  3. Goran MitrovicJanuary 18, 2013 ב 5:44 PM

    Hey, this is a coincidence – few days ago I’ve had exactly the same problem, although, platforms were a bit different (c++ dll used from InstallShield installation).

    What was especially confusing was that debug build worked, but release build hasn’t. Apparently, random garbage from the stack was incidentally zero in the case of debug build. 🙂