title: C, Open MPI: segmentation fault from call to MPI_Finalize(). Segfault does not always happen, especially with low numbers of processes
body: <p>I am writing a simple code to learn how to define an MPI_Datatype and use it in conjunction with MPI_Gatherv. I wanted to make sure I could combine variable length, dynamically allocated arrays of structured data on a process, which seems to be working fine, up until my call to MPI_Finalize(). I have confirmed that this is where the problem starts to manifest itself by using print statements and the Eclipse PTP debugger (backend is gdb-mi). My main question is, how can I get rid of the segmentation fault?</p>
  
  <p>The segfault does not occur every time I run the code. For instance, it hasn't happened for 2 or 3 processes, but tends to happen regularly when I run with about 4 or more processes. </p>
  
  <p>Also, when I run this code with valgrind, the segmentation fault does not occur. However, I do get error messages from valgrind, though the output is difficult for me to understand when I use MPI functions, even with a large number of targeted suppressions. I am also concerned that if I use more suppressions, I will silence a useful error message.</p>
  
  <p>I compile the normal code using these flags, so I am using the C99 standard in both cases:
          -ansi -pedantic -Wall -O2 -march=barcelona -fomit-frame-pointer -std=c99
  and the debugged code with:
          -ansi -pedantic -std=c99 -Wall -g</p>
  
  <p>Both use the gcc 4.4 mpicc compiler, and are run on a cluster using Red Hat Linux with Open MPI v1.4.5. Please let me know if I have left out other important bits of information. Here is the code, and thanks in advance:</p>
  
  <pre><code>//#include &lt;unistd.h&gt;
  #include &lt;string.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;math.h&gt;
  #include &lt;stdlib.h&gt;
  //#include &lt;limits.h&gt;
  
  #include "mpi.h"
  
  #define FULL_PROGRAM        1
  
  struct CD{
  
      int int_ID;
      double dbl_ID;
  };
  
  int main(int argc, char *argv[]) {
  
      int numprocs, myid, ERRORCODE;
  
  #if FULL_PROGRAM
      struct CD *myData=NULL;             //Each process contributes an array of data, comprised of 'struct CD' elements
      struct CD *allData=NULL;            //root will dynamically allocate this array to store all the data from rest of the processes
      int *p_lens=NULL, *p_disp=NULL;     //p_lens stores the number of elements in each process' array, p_disp stores the displacements in bytes
      int MPI_CD_size;                    //stores the size of the MPI_Datatype that is defined to allow communication operations using 'struct CD' elements
  
      int mylen, total_len=0;             //mylen should be the length of each process' array
                                          //MAXlen is the maximum allowable array length
                                          //total_len will be the sum of mylen across all processes
  
      // ============ variables related to defining new MPI_Datatype at runtime ====================================================
      struct CD sampleCD = {.int_ID=0, .dbl_ID=0.0};
      int blocklengths[2];                //this describes how many blocks of identical data types will be in the new MPI_Datatype
      MPI_Aint offsets[2];                //this stores the offsets, in bytes(bits?), of the blocks from the 'start' of the datatype
      MPI_Datatype block_types[2];        //this stores which built-in data types the blocks are comprised of
      MPI_Datatype  myMPI_CD;             //just the name of the new datatype
      MPI_Aint myStruct_address, int_ID_address, dbl_ID_address, int_offset, dbl_offset;  //useful place holders for filling the arrays above
      // ===========================================================================================================================
  #endif
      // =================== Initializing MPI functionality ============================
      MPI_Init(&amp;argc, &amp;argv);
      MPI_Comm_size(MPI_COMM_WORLD, &amp;numprocs);
      MPI_Comm_rank(MPI_COMM_WORLD, &amp;myid);
      // ===============================================================================
  #if FULL_PROGRAM
      // ================== This part actually formally defines the MPI datatype ===============================================
      MPI_Get_address(&amp;sampleCD, &amp;myStruct_address);          //starting point of struct CD
      MPI_Get_address(&amp;sampleCD.int_ID, &amp;int_ID_address);     //starting point of first entry in CD
      MPI_Get_address(&amp;sampleCD.dbl_ID, &amp;dbl_ID_address);     //starting point of second entry in CD
      int_offset = int_ID_address - myStruct_address;         //offset from start of first to start of CD
      dbl_offset = dbl_ID_address - myStruct_address;         //offset from start of second to start of CD
  
      blocklengths[0]=1;  blocklengths[1]=1;                  //array telling it how many blocks of identical data types there are, and the number of entries in each block
      //This says there are two blocks of identical data-types, and both blocks have only one variable in them
  
      offsets[0]=int_offset;  offsets[1]=dbl_offset;          //the first block starts at int_offset, the second block starts at dbl_offset (from 'myData_address'
  
      block_types[0]=MPI_INT; block_types[1]=MPI_DOUBLE;      //the first block contains MPI_INT, the second contains MPI_DOUBLE
  
      MPI_Type_create_struct(2, blocklengths, offsets, block_types, &amp;myMPI_CD);       //this uses the above arrays to define the MPI_Datatype...an MPI-2 function
  
      MPI_Type_commit(&amp;myMPI_CD);     //this is the final step to defining/reserving the data type
      // ========================================================================================================================
  
      mylen   = myid*2;       //each process is told how long its array should be...I used to define that randomly but that just makes things messier
  
      p_lens  = (int*)        calloc((size_t)numprocs,    sizeof(int));       //allocate memory for the number of elements (p_lens) and offsets from the start of the recv buffer(d_disp)
      p_disp  = (int*)        calloc((size_t)numprocs,    sizeof(int));
  
      myData  = (struct CD*)  calloc((size_t)mylen,       sizeof(struct CD));         //allocate memory for each process' array
      //if mylen==0, 'a unique pointer to the heap is returned'
  
      if(!p_lens) {   MPI_Abort(MPI_COMM_WORLD, 1); exit(EXIT_FAILURE);   }
      if(!p_disp) {   MPI_Abort(MPI_COMM_WORLD, 1); exit(EXIT_FAILURE);   }
      if(!myData) {   MPI_Abort(MPI_COMM_WORLD, 1); exit(EXIT_FAILURE);   }
  
  
      for(double temp=0.0;temp&lt;1e6;++temp) temp += exp(-10.0);
      MPI_Barrier(MPI_COMM_WORLD);                                //purely for keeping the output organized by give a delay in time
  
      for (int k=0; k&lt;numprocs; ++k) {
  
          if(myid==k) {
  
              //printf("\t ID %d has %d entries: { ", myid, mylen);
  
              for(int i=0; i&lt;mylen; ++i) {
  
                  myData[i]= (struct CD) {.int_ID=myid*(i+1), .dbl_ID=myid*(i+1)};            //fills data elements with simple pattern
                  //printf("%d: (%d,%lg) ", i, myData[i].int_ID, myData[i].dbl_ID);
              }
              //printf("}\n");
          }
      }
  
      for(double temp=0.0;temp&lt;1e6;++temp) temp += exp(-10.0);
      MPI_Barrier(MPI_COMM_WORLD);                            //purely for keeping the output organized by give a delay in time
  
      MPI_Gather(&amp;mylen,  1, MPI_INT, p_lens, 1, MPI_INT, 0, MPI_COMM_WORLD);     //Each process sends root the length of the vector they'll be sending
  
  #if 1
      MPI_Type_size(myMPI_CD, &amp;MPI_CD_size);          //gets the size of the MPI_Datatype for p_disp
  #else
      MPI_CD_size = sizeof(struct CD);                //using this doesn't change things too much...
  #endif
  
      for(int j=0;j&lt;numprocs;++j) {
  
          total_len += p_lens[j];
  
          if (j==0)   {   p_disp[j] = 0;                                      }
          else        {   p_disp[j] = p_disp[j-1] + p_lens[j]*MPI_CD_size;    }
      }
  
      if (myid==0)    {
  
          allData = (struct CD*)  calloc((size_t)total_len,   sizeof(struct CD));     //allocate array
          if(!allData)    {   MPI_Abort(MPI_COMM_WORLD, 1); exit(EXIT_FAILURE);   }
      }
  
      MPI_Gatherv(myData, mylen, myMPI_CD, allData, p_lens, p_disp, myMPI_CD, 0, MPI_COMM_WORLD); //each array sends root process their array, which is stored in 'allData'
  
      // ============================== OUTPUT CONFIRMING THAT COMMUNICATIONS WERE SUCCESSFUL=========================================
      if(myid==0) {
  
          for(int i=0;i&lt;numprocs;++i) {
              printf("\n\tElements from %d on MASTER are: { ",i);
              for(int k=0;k&lt;p_lens[i];++k)    {   printf("%d: (%d,%lg) ", k, (allData+p_disp[i]+k)-&gt;int_ID, (allData+p_disp[i]+k)-&gt;dbl_ID);   }
  
              if(p_lens[i]==0) printf("NOTHING ");
              printf("}\n");
          }
          printf("\n");       //each data element should appear as two identical numbers, counting upward by the process ID
      }
      // ==========================================================================================================
  
      if (p_lens) {   free(p_lens);   p_lens=NULL;    }       //adding this in didn't get rid of the MPI_Finalize seg-fault
      if (p_disp) {   free(p_disp);   p_disp=NULL;    }
      if (myData) {   free(myData);   myData=NULL;    }
      if (allData){   free(allData);  allData=NULL;   }       //the if statement ensures that processes not allocating memory for this pointer don't free anything
  
      for(double temp=0.0;temp&lt;1e6;++temp) temp += exp(-10.0);
      MPI_Barrier(MPI_COMM_WORLD);                            //purely for keeping the output organized by give a delay in time
      printf("ID %d: I have reached the end...before MPI_Type_free!\n", myid);
  
      // ====================== CLEAN UP ================================================================================
      ERRORCODE = MPI_Type_free(&amp;myMPI_CD);           //this frees the data type...not always necessary, but a good practice
  
      for(double temp=0.0;temp&lt;1e6;++temp) temp += exp(-10.0);
      MPI_Barrier(MPI_COMM_WORLD);                                //purely for keeping the output organized by give a delay in time
  
      if(ERRORCODE!=MPI_SUCCESS)  {   printf("ID %d...MPI_Type_free was not successful\n", myid); MPI_Abort(MPI_COMM_WORLD, 911); exit(EXIT_FAILURE); }
      else                        {   printf("ID %d...MPI_Type_free was successful, entering MPI_Finalize...\n", myid);       }
  #endif
      ERRORCODE=MPI_Finalize();
  
      for(double temp=0.0;temp&lt;1e7;++temp) temp += exp(-10.0);        //NO MPI_Barrier AFTER MPI_Finalize!
  
      if(ERRORCODE!=MPI_SUCCESS)  {   printf("ID %d...MPI_Finalize was not successful\n", myid);  MPI_Abort(MPI_COMM_WORLD, 911); exit(EXIT_FAILURE); }
      else                        {   printf("ID %d...MPI_Finalize was successful\n", myid);      }
  
      return EXIT_SUCCESS;
  }
  </code></pre>