title: Debug Managed Heap Corruption
body: <p>I have an interesting issue where I have two dumps of two processes which show a managed heap corruption. I am using clr.dll 4.0.30319.1008 (RTMGDR.030319-1000) in x64 on Windows 7 x64.
  With VerifyHeap I know that I have a corruption:</p>
  
  <pre><code>0:016&gt; !VerifyHeap
  object 000000000367ec60: bad member 0000000004fba740 at 000000000367ec78
  curr_object:      000000000528CF90
  Last good object: 000000000367ec40
  </code></pre>
  
  <p>The object is an array with two elements</p>
  
  <pre><code>0:016&gt; !DumpObj /d 000000000367ec60
  Name:        System.Object[]
  MethodTable: 000007feedf6adf8
  EEClass:     000007feedaefc68
  Size:        48(0x30) bytes
  Array:       Rank 1, Number of elements 2, Type CLASS (Print Array)
  Element Type:System.Object
  Fields:
  None
  
  0:016&gt; !DumpArray /d 000000000367ec60
  Name:        System.Object[]
  MethodTable: 000007feedf6adf8
  EEClass:     000007feedaefc68
  Size:        48(0x30) bytes
  Array:       Rank 1, Number of elements 2, Type CLASS
  Element Methodtable: 000007feedf65a48
  [0] 0000000004fba740
  [1] 000000000367ec90
  </code></pre>
  
  <p>The first pointer is the corrupted value it does point to a bool value with value 1 which is not a managed object. This is the reason why the GC does bail out. </p>
  
  <pre><code>0:016&gt; db 0000000004fba740-10
  00000000`04fba730  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
  00000000`04fba740  **01 00** 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
  00000000`04fba750  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
  00000000`04fba760  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
  00000000`04fba770  00 00 00 00 00 00 00 00-b8 1b f7 ed fe 07 00 00  ................
  00000000`04fba780  d0 a7 fb 04 00 00 00 00-00 00 00 00 00 00 00 00  ................
  00000000`04fba790  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
  00000000`04fba7a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
  
  0:016&gt; !lno 04fba740 
  Before:  0000000004fba718 System.Collections.Hashtable+bucket[] 
  After:   0000000004fba778 System.Collections.Hashtable 
  Heap local consistency confirmed.
  </code></pre>
  
  <p>The objects around are of no importance since depending on the dump they do change randomly. </p>
  
  <pre><code>!GCRoot 0000000367ec60
  Scan Thread 16 OSTHread 5fd0
  r10:Root:  000000000367ec60(System.Object[])
  Scan Thread 17 OSTHread 10cc
  RSP:1de4cd58:Root:  000000000367ec60(System.Object[])
  </code></pre>
  
  <p>The array itself is not rooted which indicates that it can be collected. The interesting thing is that the second object in the array is a ThreadLocal data from an already exited thread. It does look like that the CLR does store ThreadLocal objects in an object array for each thread which can be collected when it has exited. 
  Thread number 17 does the actual collection which does throw the ExecutionEngineException. But Thread 16 does seem to hold Thread Local data to an array which should be pinned (it is not) it should have no access to. </p>
  
  <p>Thread nr 16 seems to hold TLS data of an already exited thread and does perhaps write to it.</p>
  
  <pre><code>OS Thread Id: 0x5fd0 (16)
  Child SP         IP               Call Site
  000000001dffdfe8 0000000076eb135a [NDirectMethodFrameStandalone: 000000001dffdfe8] MS.Win32.UnsafeNativeMethods.MsgWaitForMultipleObjects(Int32, IntPtr[], Boolean, Int32, Int32)
  000000001dffdfa0 000007fecfa7e1bd DomainBoundILStubClass.IL_STUB_PInvoke(Int32, IntPtr[], Boolean, Int32, Int32)*** WARNING: Unable to verify checksum for UIAutomationClientsideProviders.ni.dll
  
  000000001dffe090 000007fecfa7b28d MS.Internal.AutomationProxies.Misc.MsgWaitForMultipleObjects(Microsoft.Win32.SafeHandles.SafeWaitHandle, Boolean, Int32, Int32)
  000000001dffe110 000007fecfab5cdd MS.Internal.AutomationProxies.QueueProcessor.WaitForWork()
  000000001dffe1b0 000007feede22f78 System.Threading.ExecutionContext.runTryCode(System.Object)*** WARNING: Unable to verify checksum for mscorlib.ni.dll
  
  000000001dffe8d8 000007fef08044c4 [HelperMethodFrame_PROTECTOBJ: 000000001dffe8d8] System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)
  000000001dffea00 000007feede11661 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
  000000001dffea60 000007feede115ab System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
  000000001dffeab0 000007feedea6d8d System.Threading.ThreadHelper.ThreadStart()
  000000001dffef08 000007fef08044c4 [GCFrame: 000000001dffef08] 
  000000001dfff2f0 000007fef08044c4 [DebuggerU2MCatchHandlerFrame: 000000001dfff2f0] 
  </code></pre>
  
  <p>Here is the stack of the GC collect:</p>
  
  <pre><code>0:017&gt; !DumpStack
  OS Thread Id: 0x10cc (17)
  Current frame: clr!WKS::gc_heap::mark_object_simple+0x75
  Child-SP         RetAddr          Caller, Callee
  000000001de4cce0 000007fef0877fb2 clr!WKS::gc_heap::mark_through_cards_for_segments+0x36b
  000000001de4ce50 000007fef0873980 clr!WKS::gc_heap::mark_phase+0x160, calling clr!WKS::gc_heap::mark_through_cards_for_segments
  000000001de4ce80 000007fef086fce7 clr!EEJitManager::CleanupCodeHeaps+0x57, calling clr!CrstBase::Leave
  000000001de4cea0 000007fef07e3dc1 clr!CrstBase::Leave+0x31, calling clr!GetThread
  000000001de4ced0 000007fef0873f3d clr!WKS::gc_heap::gc1+0xae, calling clr!WKS::gc_heap::mark_phase
  000000001de4cef0 000007fef0874786 clr!WKS::gc_heap::update_collection_counts+0x16, calling 000000000065006e
  000000001de4cf20 000007fef0a1fa56 clr!WKS::gc_heap::garbage_collect+0x42e, calling clr!WKS::gc_heap::gc1
  000000001de4cf60 000007feede2d774 (MethodDesc 000007feedaa93b8 +0x124 System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc(System.DateTime, Boolean ByRef)), calling (MethodDesc 000007feedaa8708 +0 System.TimeSpan.Add(System.TimeSpan))
  000000001de4cfa0 000007fef07fd4ff clr!SystemNative::__GetSystemTimeAsFileTime+0xf, calling kernel32!GetSystemTimeAsFileTimeStub
  000000001de4cff0 000007fef087452e clr!WKS::GCHeap::GarbageCollectGeneration+0x14e, calling clr!WKS::gc_heap::garbage_collect
  000000001de4d040 000007fef08734ce clr!WKS::gc_heap::try_allocate_more_space+0x25f, calling clr!WKS::GCHeap::GarbageCollectGeneration
  000000001de4d080 000007fef0872f43 clr!WKS::gc_heap::allocate_small+0x158, calling clr!WKS::gc_heap::a_fit_segment_end_p
  000000001de4d110 000007fef08731fe clr!FastAllocateObject+0x73e, calling clr!WKS::gc_heap::try_allocate_more_space
  000000001de4d1f0 000007fef07fc8b8 clr!JIT_NewFast+0xb8, calling clr!FastAllocateObject
  000000001de4d2c8 000007feede3fa80 (MethodDesc 000007feedaaa8e8 +0x40 System.Text.StringBuilder.ExpandByABlock(Int32)), calling clr!JIT_TrialAllocSFastMP_InlineGetThread
  
  0:016&gt; !Threads
  ThreadCount:      17
  UnstartedThread:  0
  BackgroundThread: 13
  PendingThread:    0
  DeadThread:       1
  Hosted Runtime:   no
                                             PreEmptive                                                   Lock
         ID  OSID        ThreadOBJ     State GC       GC Alloc Context                  Domain           Count APT Exception
     0    1  58e4 0000000000498ba0   2006020 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 STA
     2    2  4190 000000000049ee80      b220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA (Finalizer)
     6    3  48d4 000000001ac8bb60   1000220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 Ukn (Threadpool Worker)
     8    5  5fbc 000000001aca1970   a009220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA (Threadpool Completion Port)
     9    6  615c 000000001c4b2880      b020 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA
    10    7  5818 000000001c4e7bd0   200b220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA
    11    8  6e14 000000001c4f0850      7020 Enabled  0000000000000000:0000000000000000 0000000000481df0     2 STA
    12    a  683c 000000001c512610      7220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 STA
    14    b  6f40 000000001c521120      7220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 STA
    15    c  5070 000000001c564760   100a220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA (Threadpool Worker)
    16    d  5fd0 000000000049bc10      b220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA
    17    e  10cc 000000001c62e370      b220 Enabled  0000000000000000:0000000000000000 0000000000481df0     2 MTA (GC) System.ExecutionEngineException (0000000002441228)
  XXXX    f       000000001e102c80     15820 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 Ukn
    22   10  158c 000000001e103aa0   1009220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA (Threadpool Worker)
    23   12  47e8 000000001e1048c0   8019220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 Ukn (Threadpool Completion Port)
    24    4  58a8 000000001e103390   8019220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 Ukn (Threadpool Completion Port)
    25    9  2874 000000001e102570   8009220 Enabled  0000000000000000:0000000000000000 0000000000481df0     0 MTA (Threadpool Completion Port)
  </code></pre>
  
  <p>This is all nice and interesting but I am not sure how to proceed further. Since the error does occur on an automated test machine where the test controler process dies about 1-2 times per day I cannot simply attach a debugger to the process and set some breakpoints to guard writes to specific memory locations. Any additional hints how to appraoach this is highly welcome. I will get more dumps to be able to do at least a differential analysis to check which tests could cause this. </p>
  
  <p>To me it does look like that the CLR array which does hold thread statics is unpinned and someone does write an unboxed bool value to the first array element to it. The CLR array does not contain the value but the address at which normally a managed object should be but there is only the bool value (one) and not a usual CLR object with its object header. </p>
  
  <p>Could wrong PInvoke signatures cause this behaviour? I have seen some things like</p>
  
  <pre><code>    [DllImport( "kernel32.dll" )]
      public static extern bool Beep( int frequeny_in, int time_in );
  </code></pre>
  
  <p>which does return a one byte bool but the Beep Method does return a 4 byte bool. Could a wrong return type from a PInvoke (bool instead of int) cause such issues?</p>