title: Combine mpi with openMP
body: <p>I am trying to solve a problem with MPI, openMP and the combination of those two. Both MPI and openMP are running fine but when I try the MPI-openMP version of the code I can't get pass an error shows up at the end of the post.</p>
  <h2>MPI version</h2>
  <pre><code>for(i=0;i&lt;loop;i++)               
  {
    for(j=start;j&lt;end+1;j++)        
    {
      for(k=0;k&lt;N;k++)              
      {
        if(j!=k)
        {
          dx=C[k*3+0]-C[j*3+0];
          dy=C[k*3+1]-C[j*3+1];
          dz=C[k*3+2]-C[j*3+2];
  
          d=sqrt(pow(dx,2)+pow(dy,2)+pow(dz,2));
  
          dCube=pow(d,3);
          mSquare=pow(m,2);
  
          F[(j-start)*3+0]-=G*mSquare/dCube*dx;
          F[(j-start)*3+1]-=G*mSquare/dCube*dy;
          F[(j-start)*3+2]-=G*mSquare/dCube*dz;
        }
      }
    }
    for(j=0;j&lt;length;j++)                           
    {
      for(k=0;k&lt;3;k++)                               
      {
        a=F[j*3+k]/m;                               
        V[j*3+k]=V[j*3+k]+a*dt;                     
        Cw[j*3+k]=C[(j+start)*3+k]+V[j*3+k]*dt;     
      }
    }
    MPI_Allgatherv(&amp;Cw[0],length*3,MPI_DOUBLE,&amp;C[0],counts,displs,MPI_DOUBLE,MPI_COMM_WORLD);
  }
  </code></pre>
  <ul>
  <li><code>loop</code>: describes the amount the the process will take place</li>
  <li>As the <code>C</code> table needed hole for the right results the <code>start</code> &amp; <code>end</code> variables tell every node witch part of the table are responsible for</li>
  <li>The <code>MPI_allgather</code> at the end updates the C table with the new values stored in Cw tables before the new <code>i</code> loop starts</li>
  <li>The <code>counts</code> variable is the size of data that <code>MPI_Allgather</code> will transfer and the <code>displs</code> is in witch position in <code>C</code> every node must put the <code>Cw</code> data</li>
  </ul>
  <h2>openMP version</h2>
  <pre><code>#pragma omp parallel private(i,k,dx,dy,dz,d,dCube,mSquare,a)
  {
    for(i=0;i&lt;loop;i++)                          
    {
      #pragma omp for schedule(static)
        for(j=0;j&lt;N;j++)                         
        {
          for(k=0;k&lt;N;k++)                       
          {
            if(j!=k)
            {
              dx=C[k*3+0]-C[j*3+0];
              dy=C[k*3+1]-C[j*3+1];
              dz=C[k*3+2]-C[j*3+2];
  
              d=sqrt(pow(dx,2)+pow(dy,2)+pow(dz,2));
  
              dCube=pow(d,3);
              mSquare=pow(m,2);
  
              F[j*3+0]-=G*mSquare/dCube*dx;
              F[j*3+1]-=G*mSquare/dCube*dy;
              F[j*3+2]-=G*mSquare/dCube*dz;
            }
          }
        }
      #pragma omp for schedule(static)
        for(j=0;j&lt;N;j++)                        
        {
          for(k=0;k&lt;3;k++)                      
          {
            a=F[j*3+k]/m;                       
            V[j*3+k]=V[j*3+k]+a*dt;             
            C[j*3+k]=C[j*3+k]+V[j*3+k]*dt;      
          }
        }
    }
  }
  </code></pre>
  <p>The <code>i</code> loop cannot be parallel because of the need of an updated <code>C</code> in every loop so the parallel part starts from the <code>j</code></p>
  <h2>Combined version</h2>
  <pre><code>#pragma omp parallel private(i,k,dx,dy,dz,d,dCube,mSquare,a)
    {
      for(i=0;i&lt;loop;i++)                                 
      {
        #pragma omp for schedule(static)
          for(j=start;j&lt;end+1;j++)                        
          {
            for(k=0;k&lt;N;k++)                              
            {
              if(j!=k)
              {
                dx=C[k*3+0]-C[j*3+0];
                dy=C[k*3+1]-C[j*3+1];
                dz=C[k*3+2]-C[j*3+2];
  
                d=sqrt(pow(dx,2)+pow(dy,2)+pow(dz,2));
  
                dCube=pow(d,3);
                mSquare=pow(m,2);
  
                F[(j-start)*3+0]-=G*mSquare/dCube*dx;
                F[(j-start)*3+1]-=G*mSquare/dCube*dy;
                F[(j-start)*3+2]-=G*mSquare/dCube*dz;
              }
            }
          }
        #pragma omp for schedule(static)
          for(j=0;j&lt;length;j++)                           
          {
            for(k=0;k&lt;3;k++)                              
            {
              a=F[j*3+k]/m;                               
              V[j*3+k]=V[j*3+k]+a*dt;                     
              Cw[j*3+k]=C[(j+start)*3+k]+V[j*3+k]*dt;     
            }
          }
        MPI_Allgatherv(&amp;Cw[0],length*3,MPI_DOUBLE,&amp;C[0],counts,displs,MPI_DOUBLE,MPI_COMM_WORLD);
      }
    }
  </code></pre>
  <p>What I've tried to achieve here with <code>MPI</code> is to say in every node with witch part of the <code>C</code> to work on as the MPI approach above and then openMP calculates the <code>Cw</code> tables correspond to this part. Then MPI gather the results with <code>MPI_Allgather</code> in the <code>C</code> table and starts over for <code>loop</code> times. The problem is that when I run the code above this error shows up:</p>
  <blockquote>
  <p>Assertion failed in file src/mpid/ch3/src/ch3u_request.c at line 572: *(&amp;incomplete) &gt;= 0</p>
  <p>=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES <br/>
  =   PID 10389 RUNNING AT felix-desktop <br/>
  =   EXIT CODE: 139 <br/>
  =   CLEANING UP REMAINING PROCESSES <br/>
  =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES <br/></p>
  <p>YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11) <br/>
  This typically refers to a problem with your application. <br/>
  Please see the FAQ page for debugging suggestions <br/></p>
  </blockquote>
  <p>The problem lies int the <code>MPI_Allgather</code> line of code but I can't figure what is the cause of it.</p>