title: Post-mortem debugging of a Mono AOT-compiled program
body: <p>I have a large program, written in C# and running on Linux systems using Mono, that occasionally crashes and causes the <code>mono.bin</code> process to dump core.</p>
  
  <p>I ran <code>gdb</code> on some of the core dump files, but it wasn't very useful because the backtraces don't have the names of the C# functions in them.  According to <a href="http://mono.1490590.n4.nabble.com/Post-mortem-debugging-td4238349.html" rel="nofollow noreferrer">this discussion I found</a>:</p>
  
  <blockquote>
    <p>It won't work. The information required to construct managed stack
    traces is contained in runtime data structures, and it is only
    available while the program is running. You can AOT your application,
    then you will have more usable stack traces.</p>
  </blockquote>
  
  <p>So, I did.  I AOT-compiled all of my C# DLL and EXE files.  Using the <code>--aot=write-symbols</code> option.  For a test version of my program that <a href="https://stackoverflow.com/questions/3766122/what-is-a-canonical-way-to-produce-a-segmentation-fault-in-c">crashes on purpose</a> so I could check if this makes the backtraces more useful.  And so far, it hasn't.  The backtrace from the main thread looks like:</p>
  
  <pre><code>#0  0xb7fc8402 in __kernel_vsyscall ()
  #1  0x00556df0 in raise () from /lib/libc.so.6
  #2  0x00558701 in abort () from /lib/libc.so.6
  #3  0x080e59b5 in ?? ()
  </code></pre>
  
  <p>Another thread has:</p>
  
  <pre><code>#0  0xb7fc8402 in __kernel_vsyscall ()
  #1  0x005f6753 in poll () from /lib/libc.so.6
  #2  0xb6f735a7 in Mono_Unix_UnixSignal_WaitAny ()
     from /opt/novell/mono/lib/libMonoPosixHelper.so
  #3  0xb5416578 in ?? ()
  </code></pre>
  
  <p>And other threads seem to have been idle (in <code>nanosleep</code>, <code>pthread_cond_timedwait</code>, <code>pthread_cond_wait</code>, <code>sem_timedwait</code>, or <code>sem_wait</code>).  But the thing all backtraces have in common is that they end with that annoying <code>in ?? ()</code>, and never list any function names from "my" code.</p>
  
  <p>I think this is related to some messages that <code>gdb</code> printed when it started up; for example,</p>
  
  <pre><code>Reading symbols from /xyz/mono/log4net.dll.so...(no debugging symbols found)...done.
  Loaded symbols for /xyz/mono/log4net.dll.so
  Reading symbols from /xyz/mono/Contoso.Util.dll.so...(no debugging symbols found)...done.
  Loaded symbols for /xyz/mono/Contoso.Util.dll.so
  Reading symbols from /xyz/mono/Contoso.Printing.dll.so...(no debugging symbols found)...done.
  Loaded symbols for /xyz/mono/Contoso.Printing.dll.so
  Reading symbols from /xyz/mono/Contoso.LegacyDataConverter.dll.so...(no debugging symbols found)...done.
  Loaded symbols for /xyz/mono/Contoso.LegacyDataConverter.dll.so
  </code></pre>
  
  <p>Why do all the <code>*.dll.so</code> files have "no debugging symbols found"?  Do the DLLs themselves need to be built in "debug" mode or something?</p>
  
  <p>And more generally, is there a way to get the managed stack trace from a Mono core dump?  (Without using <code>mono_pmip</code>, because that's only available when the process is running.)</p>