title: Debugging a memory leak in a long-running mono/C# application
body: <p>I have a C# program that runs on an Ubuntu VM as a server using mono 5.10. Every now and then, the program starts using lots of memory until it eventually crashes. It can often take days before such a memory leak occurs, and I haven't been able to reproduce the issue locally.</p>
  
  <p>For the past few days, I've used the mono log profiler to be able to collect heapshots on demand. This has given me a 15Gb+ file. I managed to get a heapshot from when the memory leak occurs, which displays as follows in the Xamarin profiler:</p>
  
  <p><a href="https://i.stack.imgur.com/XMCtS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XMCtS.png" alt="HeapShot summary in the Xamarin profiler"></a></p>
  
  <p>Running <code>mprof-report</code> gives a similarly unhelpful report.</p>
  
  <p>Neither really help debug the problem. Obviously there's something going on with the threadpool, but it seems like there is no way of figuring out what.</p>
  
  <p>Being able to see where the objects are allocated might help, but that requires enabling <code>alloc</code> in the profiler, which is not possible because of the file size it will produce.</p>
  
  <p>What is causing the massive amount of <code>IThreadPoolWorkItem</code>s? The app is using 30-40% CPU, so it doesn't seem like tasks are being scheduled more quickly than the CPU can handle.</p>
  
  <p>Is there a way to list the objects referenced by the ThreadPool jobs? That would at least allow me to identify what piece of code is being run so much. <code>mprof-report</code> only shows inverse references as far as I can see, which isn't very helpful as the <code>IThreadPoolWorkItem</code>s are obviously owned by the <code>ThreadPool</code> itself.</p>
  
  <p><strong>Update:</strong> The tasks aren't (disk) IO bound. While leaking, the reads are in the 10s of kbs per second, which shouldn't be saturating the disk.</p>
  
  <p>On second thought: if there were a lot of tasks scheduled, shouldn't I be seeing lots of individual <code>IThreadPoolWorkItem</code> objects too? Since <code>IThreadPoolWorkItem</code> isn't a struct, there should be a separate entry for the actual object and <code>IThreadPoolWorkItem[]</code> would just be an array of pointers. So this would suggest that those <code>IThreadPoolWorkItem[]</code>s are just arrays of (mostly) <code>null</code> pointers.</p>
  
  <p><strong>Update:</strong> <code>ThreadPool.GetAvailableThreads</code> shows 2 worker threads and 0 completion port threads are active, which seems to indicate that the <code>ThreadPool</code> is doing just fine. Custom logging also indicates that once again, I'm not scheduling too many tasks and the tasks that are scheduled are finished quickly.</p>