title: Wait for all jobs of a user to finish before submitting subsequent jobs to a PBS cluster
body: <p>I am trying to adjust some bash scripts to make them run on a (<a href="/questions/tagged/pbs" class="post-tag" title="show questions tagged &#39;pbs&#39;" rel="tag">pbs</a>) cluster.</p>
  
  <p>The individual tasks are performed by several script thats are started by a main script.
  So far this main scripts starts multiple scripts in background (by appending <code>&amp;</code>) making them run in parallel on one multi core machine.
  I want to substitute these calls by <code>qsub</code>s to distribute load accross the cluster nodes.</p>
  
  <p>However, some jobs depend on others to be finished before they can start.
  So far, this was achieved by <code>wait</code> statements in the main script.
  But what is the best way to do this using the grid engine?</p>
  
  <p>I already found <a href="https://stackoverflow.com/questions/11525214/wait-for-set-of-qsub-jobs-to-complete">this question</a> as well as the <code>-W after:jobid[:jobid...]</code> documentation in the <code>qsub</code> man page but I hope there is a better way.
  We are talking about several thousend jobs to run in parallel first and another set of the same size to run simultatiously after the last one of these finished.
  This would mean I had to queue a lot of jobs depending on a lot of jobs.</p>
  
  <p>I could bring this down by using a dummy job in between, doing nothing but depending on the first group of jobs, on which the second group could depend.
  This would decrease the number of dependencies from millions to thousands but still: It feeles wrong and I am not even sure if such a long command line would be accepted by the shell.</p>
  
  <ul>
  <li>Isn't there a way to wait for all <em>my</em> jobs to finish (something like <code>qwait -u &lt;user&gt;</code>)?</li>
  <li>Or all jobs that where submitted from <em>this</em> script (something like <code>qwait [-p &lt;PID&gt;]</code>)?</li>
  </ul>
  
  <p>Of course it would be possible to write something like this using <code>qstat</code> and <code>sleep</code> in a <code>while</code> loop, but I guess this use case is important enough to have a built in solution and I was just incapable to figure that one out.</p>
  
  <p><strong>What would you recommend / use in such a situation?</strong></p>
  
  <p><strong>Addendum I:</strong></p>
  
  <p>Since it was requested in a comment:</p>
  
  <pre class="lang-none prettyprint-override"><code>$ qsub --version
  version: 2.4.8
  </code></pre>
  
  <p>Maybe also helpful to determine the exact <a href="/questions/tagged/pbs" class="post-tag" title="show questions tagged &#39;pbs&#39;" rel="tag">pbs</a> system:</p>
  
  <pre class="lang-none prettyprint-override"><code>$ qsub --help
  usage: qsub [-a date_time] [-A account_string] [-b secs]
        [-c [ none | { enabled | periodic | shutdown |
        depth=&lt;int&gt; | dir=&lt;path&gt; | interval=&lt;minutes&gt;}... ]
        [-C directive_prefix] [-d path] [-D path]
        [-e path] [-h] [-I] [-j oe] [-k {oe}] [-l resource_list] [-m n|{abe}]
        [-M user_list] [-N jobname] [-o path] [-p priority] [-P proxy_user] [-q queue]
        [-r y|n] [-S path] [-t number_to_submit] [-T type] [-u user_list] [-w] path
        [-W otherattributes=value...] [-v variable_list] [-V] [-x] [-X] [-z] [script]
  </code></pre>
  
  <p>Since the comments point to job arrays so far I searched the <code>qsub</code> man page with the following results:</p>
  
  <pre class="lang-none prettyprint-override"><code>[...]
  DESCRIPTION
  [...]
         In addition to the above, the following environment variables will be available to the batch job.
  [...]
         PBS_ARRAYID
                each member of a job array is assigned a unique identifier (see -t)
  [...]
  OPTIONS
  [...]
         -t array_request
                 Specifies the task ids of a job array. Single task arrays are allowed.
                 The array_request argument is an integer id or a range of integers. Multiple ids or id ranges can be combined in a comman delimeted list. Examples : -t 1-100 or -t 1,10,50-100
  [...]
  </code></pre>
  
  <p><strong>Addendum II:</strong></p>
  
  <p>I have tried the <a href="/questions/tagged/torque" class="post-tag" title="show questions tagged &#39;torque&#39;" rel="tag">torque</a> solution given by Dmitri Chubarov but it does not work as described.</p>
  
  <p>Without the job arrray it works as expected:</p>
  
  <pre><code>testuser@headnode ~ $ qsub -W depend=afterok:`qsub ./test1.sh` ./test2 &amp;&amp; qstat
  2553.testserver.domain
  Job id                  Name             User            Time Use S Queue
  ----------------------- ---------------- --------------- -------- - -----
  2552.testserver         Test1            testuser               0 Q testqueue
  2553.testserver         Test2            testuser               0 H testqueue
  testuser@headnode ~ $ qstat
  Job id                  Name             User            Time Use S Queue
  ----------------------- ---------------- --------------- -------- - -----
  2552.testserver         Test1            testuser               0 R testqueue
  2553.testserver         Test2            testuser               0 H testqueue
  testuser@headnode ~ $ qstat
  Job id                  Name             User            Time Use S Queue
  ----------------------- ---------------- --------------- -------- - -----
  2553.testserver         Test2            testuser               0 R testqueue
  </code></pre>
  
  <p>However, using job arrays the second job won't start:</p>
  
  <pre><code>testuser@headnode ~ $ qsub -W depend=afterok:`qsub -t 1-2 ./test1.sh` ./test2 &amp;&amp; qstat
  2555.testserver.domain
  Job id                  Name             User            Time Use S Queue
  ----------------------- ---------------- --------------- -------- - -----
  2554-1.testserver       Test1-1          testuser               0 Q testqueue
  2554-2.testserver       Test1-1          testuser               0 Q testqueue
  2555.testserver         Test2            testuser               0 H testqueue
  testuser@headnode ~ $ qstat
  Job id                  Name             User            Time Use S Queue
  ----------------------- ---------------- --------------- -------- - -----
  2554-1.testserver       Test1-1          testuser               0 R testqueue
  2554-2.testserver       Test1-2          testuser               0 R testqueue
  2555.testserver         Test2            testuser               0 H testqueue
  testuser@headnode ~ $ qstat
  Job id                  Name             User            Time Use S Queue
  ----------------------- ---------------- --------------- -------- - -----
  2555.testserver         Test2            testuser               0 H testqueue
  </code></pre>
  
  <p>I guess this is due to the lack of array indication in the job id that is returned by the first <code>qsub</code>:</p>
  
  <pre><code>testuser@headnode ~ $ qsub -t 1-2 ./test1.sh
  2556.testserver.domain
  </code></pre>
  
  <p>As you can see there is no <code>...[]</code> indicating this being a job array.
  Also, in the <code>qsub</code> output there are no <code>...[]</code>s but <code>...-1</code> and <code>...-2</code> indicating the array.</p>
  
  <p>So the remaining question is how to format <code>-W depend=afterok:...</code> to make a job depend on a specified job array.</p>