title: How to hold up a script until a slurm job (start with srun) is completely finished?
body: <p>I am running a job array with SLURM, with the following job array script (that I run with <code>sbatch job_array_script.sh [args]</code>:</p>
  
  <pre><code>#!/bin/bash
  
  #SBATCH ... other options ...
  
  #SBATCH --array=0-1000%200
  
  srun ./job_slurm_script.py $1 $2 $3 $4
  
  echo 'open' &gt; status_file.txt
  </code></pre>
  
  <p>To explain, I want <code>job_slurm_script.py</code> to be run as an array job 1000 times with 200 tasks maximum in parallel.  And when <strong>all</strong> of those are done, I want to write 'open' to <code>status_file.txt</code>. This is because in reality I have more than 10,000 jobs, and this is above my cluster's MaxSubmissionLimit, so I need to split it into smaller chunks (at 1000-element job arrays) and run them one after the other (only when the previous one is finished).</p>
  
  <p>However, for this to work, the echo statement can only trigger once the entire job array is finished (outside of this, I have a loop which checks <code>status_file.txt</code> so see if the job is finished, i.e when the contents are the string 'open').</p>
  
  <p>Up to now I thought that <code>srun</code> holds the script up until the whole job array is finished. However, sometimes <code>srun</code> "returns" and the script goes to the echo statement before the jobs are finished, so all the subsequent jobs bounce off the cluster since it goes above the submission limit.</p>
  
  <p>So how do I make <code>srun</code> "hold up" until the whole job array is finished?</p>