title: PInvoke x64 crash with .Net 4.0
body: <p>I've been tasked with getting some C# code working in x64 that calls a native x64 dll called Detagger that is used for converting HTML into Text while maintaining the basic stucture of the HTML.  </p>
  
  <p>This code has worked for years when running with platform target x86 for the C# code and an x86 build of the dll, but it's crashing when setting the platform target to x64 and using an x64 build of the dll.  In fact, x64 works fine if the C# app is built with the .Net framework 3.5 or below.  It crashes when built with 4.0 or above.</p>
  
  <p>The dll in question has the following header:
  </p>
  
  <pre><code>#ifdef WIN32
      #ifdef USE_DLL
      #ifdef DLL_EXPORTS
          #define DLL_DECLARE __declspec(dllexport) long __stdcall
      #else
          #define DLL_DECLARE __declspec(dllimport) long __stdcall
      #endif
      #else
      #define DLL_DECLARE long
      #endif
  #else
      #define DLL_DECLARE long
  #endif
  
  ...
  
  DLL_DECLARE CONVERTER_Allocate ();  // returns non-zero Handle if succeeds
  
  ...
  
  DLL_DECLARE CONVERTER_ResetPolicies (long Handle);
  </code></pre>
  
  <p>And so the API requires calling the CONVERT_Allocate() function to get a "handle" (which I think is actually a memory address) and then passing that "handle" into all of the other methods.  I presume this is for making the calls thread safe.</p>
  
  <p>I'm trying to focus on the CONVERTER_ResetPolicies() function for now, because that is one of the most basic ones that takes just a single parameter (the "handle").  None of the functions in the entire API are complicated, all taking basic types or pointers to such as parameters (no structs).</p>
  
  <p>From the C++ header, the calling convention is supposedly stdcall, and each of the exported functions in the dll returns a long (which should be 4 bytes in both x86 and x64).  My understanding of x64 is that its calling convention is basically always a variant of fastcall, so I'm curious about the stdcall, but it works in .Net 3.5 and below so that's a question for another day.</p>
  
  <p>The PInvoke signatures provided by the vendor for the dll are:
  </p>
  
  <pre><code>// DLL_DECLARE CONVERTER_Allocate();
  [DllImport(_dll, EntryPoint = "CONVERTER_Allocate")]
  public static extern IntPtr Allocate();
  
  // DLL_DECLARE CONVERTER_ResetPolicies(long Handle);
  [DllImport(_dll, EntryPoint = "CONVERTER_ResetPolicies")]
  public static extern APIResult ResetPolicies(IntPtr handle);
  </code></pre>
  
  <p>Given the following C# code:</p>
  
  <pre><code>IntPtr handle = DetaggerAPI.Allocate();
  var result = DetaggerAPI.ResetPolicies();
  </code></pre>
  
  <p>This crashes in the call to CONVERTER_ResetPolicies().  Stepping in the debugger reveals the following:</p>
  
  <p>In C#:
      handle = 0x00000000e82d0080</p>
  
  <p>In disassembly after stepping into the DLL:</p>
  
  <p>registers and flags:</p>
  
  <pre><code>RAX = 000000018001B490 RBX = 0000000FCC66EB68 RCX = 00000000E82D0080
  RDX = 0000000FCC66EC80 RSI = 0000000FCF8B44A8 RDI = 0000000FCC66E980 
  R8  = 00001EB6102A86D4 R9  = 0000000FE84C4001 R10 = 00007FF9497961F0
  R11 = 0000000000000000 R12 = 0000000000000000 R13 = 0000000FCC66EAF0
  R14 = 0000000FCC66EB68 R15 = 0000000000000004 RIP = 000000018001B490 
  RSP = 0000000FCC66E848 RBP = 0000000FCC66E850 EFL = 00000246 
  
  CS = 0033 DS = 0000 ES = 0000 SS = 002B FS = 0000 GS = 0000 
  
  OV = 0 UP = 0 EI = 1 PL = 0 ZR = 1 AC = 0 PE = 1 CY = 0 
  </code></pre>
  
  <p>Note that the value for handle is in RCX (e82d0080).</p>
  
  <p>Here is the dissassembly (some comments added by me):</p>
  
  <pre><code>000000018001B490  sub         rsp,28h                   ; subtract 40 from stack pointer, sets up stack frame
  000000018001B494  call        000000018001B090  
  
      000000018001B090  push        rbx  
      000000018001B092  sub         rsp,20h               ; subtract 32 from stack pointer, sets up stack frame
      000000018001B096  test        ecx,ecx               ; check if ecx is 0
      000000018001B098  movsxd      rbx,ecx               ; move value in ecx (the handle passed in) to rbx and sign-extend it to qword
                                                          ; rbx changes from 0000000FCC66EB68 to FFFFFFFFE82D0080
      000000018001B09B  je          000000018001B0C6      ; if ecx is 0, probably jump to a function that returns an error
  -&gt;  000000018001B09D  cmp         dword ptr [rbx],4D2h  ; compare value pointed to by rbx (as a dword) to 042d (1234),
                                                          ; but rbx points to FFFFFFFFE82D0080, which is probably an invalid memory location,
                                                          ; so !!this is the line that crashes !!
      000000018001B0A3  jne         000000018001B0C6      ; jump if not equal
  
      000000018001B0A5  mov         ecx,dword ptr [1801122C0h]  
      000000018001B0AB  mov         dword ptr [rbx+2F0B0h],ecx  
      000000018001B0B1  lea         rcx,[rbx+2F0B8h]  
      000000018001B0B8  call        00000001800A7C40  
      000000018001B0BD  mov         rax,rbx  
      000000018001B0C0  add         rsp,20h  
      000000018001B0C4  pop         rbx  
      000000018001B0C5  ret  
  
  000000018001B499  test        rax,rax  
  000000018001B49C  jne         000000018001B4BC  
  000000018001B49E  cmp         dword ptr [1801122C0h],eax  
  000000018001B4A4  je          000000018001B4B2  
  000000018001B4A6  lea         rcx,[1800D7B70h]  
  000000018001B4AD  call        000000018001B290  
  000000018001B4B2  mov         eax,2                     ; if we got here, return 2 in eax, meaning APIResult.Invalid.  Note that this is 32bits.
  000000018001B4B7  add         rsp,28h                   ; clean up stack frame
  000000018001B4BB  ret                                   ; return
  </code></pre>
  
  <p>So, looks like the "handle" is being passed in RCX, and then subsequently the</p>
  
  <pre><code>movsxd  rbx,ecx
  </code></pre>
  
  <p>instruction is copying this handle into RBX but also basically destroying it since it appears to be a memory address rather than just some opaque handle that is an array index or something similar.  Then two instructions later I get an access violation from the instruction</p>
  
  <pre><code>cmp dword ptr [rbx],4D2h
  </code></pre>
  
  <p>because this is trying to dereference RBX, which points to garbage.</p>
  
  <p>According to <a href="https://msdn.microsoft.com/en-us/library/ee941656(v=vs.100).aspx#core" rel="nofollow">https://msdn.microsoft.com/en-us/library/ee941656(v=vs.100).aspx#core</a>, under Platform Invoke, it says the difference between 3.5 SP1 and 4.0 is:</p>
  
  <blockquote>
    <p>To improve performance in interoperability with unmanaged code,
    incorrect calling conventions in a platform invoke now cause the
    application to fail.      In previous versions, the marshaling layer
    resolved these errors up the stack.</p>
  </blockquote>
  
  <p>That is kind of vague, but since my only option here is stdcall (fastcall is not supported), I presume that is correct and not the issue.</p>
  
  <p>Some things I'm going to try:</p>
  
  <ol>
  <li>Debugging running against .Net 3.5 and try to see what's different. </li>
  <li>Create a C++/cli wrapper for the dll instead of using PInvoke.</li>
  </ol>
  
  <p>If anyone can spot what's going on here or give me any ideas, that'd be great.</p>