title: Debugging core files generated on a Customer's box
body: <p>We get core files from running our software on a Customer's box.  Unfortunately because we've always compiled with -O2 <em>without</em> debugging symbols this has lead to situations where we could not figure out why it was crashing, we've modified the builds so now they generate -g and -O2 together.  We then advice the Customer to run a -g binary so it becomes easier to debug.</p>
  
  <p>I have a few questions:</p>
  
  <ol>
  <li>What happens when a core file is generated from a Linux distro other than the one we are running in Dev?  Is the stack trace even meaningful?</li>
  <li>Are there any good books for debugging on Linux, or Solaris?  Something example oriented would be great.  I am looking for real-life examples of figuring out why a routine crashed and how the author arrived at a solution.  Something more on the intermediate to advanced level would be good, as I have been doing this for a while now.  Some assembly would be good as well.</li>
  </ol>
  
  <p>Here's an example of a crash that requires us to tell the Customer to get a -g ver. of the binary:</p>
  
  <pre><code>Program terminated with signal 11, Segmentation fault.
  #0  0xffffe410 in __kernel_vsyscall ()
  (gdb) where
  #0  0xffffe410 in __kernel_vsyscall ()
  #1  0x00454ff1 in select () from /lib/libc.so.6
  ...
  &lt;omitted frames&gt;
  </code></pre>
  
  <p>Ideally I'd like to solve find out why exactly the app crashed - I suspect it's memory corruption but I am not 100% sure.</p>
  
  <p>Remote debugging is strictly not allowed.</p>
  
  <p>Thanks</p>