title: MPI: Canceling Non-blocking Send
body: <p>I'm using Open MPI library to implement the following algorithm: we have two processes <code>p1</code> and <code>p2</code>. They're both executing some iterations and at the end of each iteration, they communicate their results. The problem is that the execution is not necessarily balanced, so <code>p1</code> may execute 10 iterations in the time <code>p2</code> executes 1. Even though, I want <code>p2</code> to read the latest result from the last iteration executed by <code>p1</code>. </p>
  
  <p>Thus, my idea is that <code>p1</code> sends its results at each iteration. But, before sending the result from an iteration <code>i</code>, it should check if <code>p2</code> actually read the information from iteration <code>i-1</code>. If not, it should cancel the previous <em>send</em> so that when <code>p2</code> reads from <code>p1</code>, it will  read the most recent result.</p>
  
  <p>Unfortunately, I'm not sure how to do that. I've tried using MPI_Cancel, as in the following code:</p>
  
  <pre><code>int main (int argc, char *argv[]){
  
      int myrank, numprocs;
      MPI_Status status;
      MPI_Request request;
  
      MPI_Init(&amp;argc, &amp;argv);
      MPI_Comm_size(MPI_COMM_WORLD, &amp;numprocs);
      MPI_Comm_rank(MPI_COMM_WORLD, &amp;myrank);
  
      if(myrank == 0){
          int send_buf = 1, flag;
          MPI_Isend(&amp;send_buf, 1, MPI_INT, 1, 123, MPI_COMM_WORLD, 
                    &amp;request);
          MPI_Cancel(&amp;request);
          MPI_Wait(&amp;request, &amp;status);
          MPI_Test_cancelled(&amp;status, &amp;flag);
          if (flag) printf("Send cancelled\n");
          else printf("Send NOT cancelled\n");
          send_buf = 2;
          MPI_Isend(&amp;send_buf, 1, MPI_INT, 1, 123, MPI_COMM_WORLD, 
                    &amp;request);
      }
      else {
          sleep(5);
          int msg;
          MPI_Recv(&amp;msg, 1, MPI_INT, 0, 123,
                   MPI_COMM_WORLD, &amp;status);
          printf("%d\n", msg);
      }
      MPI_Finalize();
  
      return 0;
  }
  </code></pre>
  
  <p>But when I execute, it says that the send could not be cancelled and <code>p2</code> prints 1 instead of 2.</p>
  
  <p>I'd like to know if there's any way to achieve what I'm proposing or if there's an alternative to code the behaviour between <code>p1</code> and <code>p2</code>.</p>