title: MPI and Valgrind not showing line numbers
body: <p>I've written a large program and I'm having a really hard time tracking down a segmentation fault. I posted a question but I didn't have enough information to go on (see link below - and if you do, note that I spent almost an entire day trying several times to come up with a minimally compilable version of the code that reproduced the error to no avail).</p>
  
  <p><a href="https://stackoverflow.com/questions/16025411/phantom-bug-involving-stdvectors-mpi-c">https://stackoverflow.com/questions/16025411/phantom-bug-involving-stdvectors-mpi-c</a></p>
  
  <p>So now I'm trying my hand at valgrind for the first time. I just installed it (simply "sudo apt-get install valgrind") with no special installation to account for MPI (if there is any). I'm hoping for concrete information including file names and line numbers (I understand it's impossible for valgrind to provide variable names). While I am getting useful information, including</p>
  
  <ul>
  <li>Invalid read of size 4</li>
  <li>Conditional jump or move depends on uninitialised value(s)</li>
  <li>Uninitialised value was created by a stack allocation</li>
  <li>4 bytes in 1 blocks are definitely lost</li>
  </ul>
  
  <p>in addition to this magical thing</p>
  
  <ul>
  <li>Syscall param sched_setaffinity(mask) points to unaddressable byte(s) at 0x433CE77: syscall (syscall.S:31) Address 0x0 is not stack'd, malloc'd or (recently) free'd</li>
  </ul>
  
  <p>I am not getting file names and line numbers. Instead, I get</p>
  
  <pre><code>==15095==    by 0x406909A: ??? (in /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
  </code></pre>
  
  <p>Here's how I compile my code:</p>
  
  <pre><code>mpic++ -Wall -Wextra -g -O0 -o Hybrid.out (…file names)
  </code></pre>
  
  <p>Here are two ways I've executed valgrind:</p>
  
  <pre><code>valgrind --tool=memcheck --leak-check=full --track-origins=yes --log-file=log.txt mpirun -np 1 Hybrid.out
  </code></pre>
  
  <p>and</p>
  
  <pre><code>mpirun -np 1 valgrind --tool=memcheck --leak-check=full --track-origins=yes --log-file=log4.txt -v ./Hybrid.out
  </code></pre>
  
  <p>The second version based on instructions in</p>
  
  <p><a href="https://stackoverflow.com/questions/5247526/segmentation-faults-occur-when-i-run-a-parallel-program-with-open-mpi">Segmentation faults occur when I run a parallel program with Open MPI</a></p>
  
  <p>which, if I'm understanding the chosen answer correctly, appears to be contradicted by</p>
  
  <p><a href="https://stackoverflow.com/questions/11077746/openmpi-with-valgrind-can-i-compile-with-mpi-in-ubuntu-distro">openmpi with valgrind (can I compile with MPI in Ubuntu distro?)</a></p>
  
  <p>I am deliberately running valgrind on one processor because that's the only way my program will execute to completion without the segmentation fault. I have also run it with two processors, and my program seg faulted as expected, but the log I got back from valgrind seemed to contain essentially the same information. I'm hoping that by resolving the issues valgrind reports on one processor, I'll magically solve the issue happening on more than one.</p>
  
  <p>I tried to include "-static" in the program compilation as suggested in </p>
  
  <p><a href="https://stackoverflow.com/questions/9321385/valgrind-not-showing-line-numbers-in-spite-of-g-flag-on-ubuntu-11-10-virtualbo">Valgrind not showing line numbers in spite of -g flag (on Ubuntu 11.10/VirtualBox)</a></p>
  
  <p>but the compilation failed, saying (in addition to several warnings)</p>
  
  <pre><code>dynamic STT_GNU_IFUNC symbol "strcmp" with pointer equality in '…' can not be used when making an executably; recompile with fPIE and relink with -pie
  </code></pre>
  
  <p>I have not looked into what "fPIE" and "-pie" mean. Also, please note that I am not using a makefile, nor do I currently know how to write one.</p>
  
  <p>A few more notes: My code does not use the commands malloc, calloc, or new. I'm working entirely with std::vector; no C arrays. I do use commands like .resize(), .insert(), .erase(), and .pop_back(). My code also passes vectors to functions by reference and constant reference. As for parallel commands, I only use MPI_Barrier(), MPI_Bcast(), and MPI_Allgatherv().</p>
  
  <p>How do I get valgrind to show the file names and line numbers for the errors it is reporting? Thank you all for your help!</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>I continued working on it and a friend of mine pointed out that the reports without line numbers are all coming from MPI files, which I did not compile from source, and since I did not compile them, I can't use the -g option, and hence, don't see lines. So I tried valgrind again based on this command,</p>
  
  <pre><code>mpirun -np 1 valgrind --tool=memcheck --leak-check=full --track-origins=yes --log-file=log4.txt -v ./Hybrid.out
  </code></pre>
  
  <p>but now for two processors, which is</p>
  
  <pre><code>mpirun -np 2 valgrind --tool=memcheck --leak-check=full --track-origins=yes --log-file=log4.txt -v ./Hybrid.out
  </code></pre>
  
  <p>The program ran to completion (I did not see the seg fault reported in the command line) but this execution of valgrind did give me line numbers within my files. The line valgrind is pointing to is a line where I call MPI_Bcast(). Is it safe to say that this appeared because the memory problem only manifests itself on multiple processors (since I've run it successfully on np -1)?</p>