title: Why does Slurm kill one particular user's jobs after a few seconds?
body: <p>I administer a Slurm cluster with many users and the operation of the cluster currently appears &quot;totally normal&quot; for all users; except for one.
  This one user gets all attempts to run commands through Slurm killed after 20-25 seconds.<br />
  The following minimal example reproduces the error:</p>
  <pre><code>$ sudo -u &lt;the_user&gt; srun --pty sleep 25
  srun: job 110962 queued and waiting for resources
  srun: job 110962 has been allocated resources
  srun: Force Terminated job 110962
  srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
  slurmstepd: error: *** STEP 110962.0 ON &lt;node&gt; CANCELLED AT 2021-04-09T16:33:35 ***
  srun: error: &lt;node&gt;: task 0: Terminated
  </code></pre>
  <p>When this happens, I find this line in the <code>slurmctld</code> log:</p>
  <pre><code>_slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=110962 uid &lt;the_users_uid&gt;
  </code></pre>
  <p>It only happens for '&lt;the_user&gt;' and not for any other user that I know of. This very similar but shorter-running example works fine:</p>
  <pre><code>$ sudo -u &lt;the_user&gt; srun --pty sleep 20
  srun: job 110963 queued and waiting for resources
  srun: job 110963 has been allocated resources
  </code></pre>
  <p>Note that when I run <code>srun --pty sleep 20</code> as myself, <code>srun</code> does not output the two <code>srun: job...</code> lines. This seems to me to be an additional indication that <code>srun</code> is subject to some different settings for '&lt;the_user&gt;'.<br />
  All settings that I have been able to inspect appear identical for '&lt;the_user&gt;' as for other users. I have checked, and 'MaxWall' is <em>not</em> set for this user and not for any other user, either. Other users belonging to the same Slurm account do not experience this problem.<br />
  <a href="https://stackoverflow.com/questions/63776246/srun-used-in-a-loop-srun-job-step-aborted-waiting-up-to-32-seconds-for-job-st">This question</a> sounds related, but I do not think the explanation seems to be the same.</p>
  <p>What could be causing this?</p>
  <p><strong>Update - the plot thickens</strong></p>
  <p>When this unfortunate user's jobs get allocated, I see this message in '/var/log/slurm/slurmctld.log':</p>
  <pre><code>sched: _slurm_rpc_allocate_resources JobId=111855 NodeList=&lt;node&gt;
  </code></pre>
  <p>and shortly after, I see this message:</p>
  <pre><code>select/cons_tres: common_job_test: no job_resources info for JobId=110722_* rc=0
  </code></pre>
  <p>Job 110722_* is a pending array job by another user that is pending due to 'QOSMaxGRESPerUser'. One pending part of this array job (110722_57) eventually ends up taking over job 111855's CPU cores when 111855 gets killed. This leads me to believe that 110722_57 causes 111855 to be killed. However, 110722_57 remains pending afterwards.</p>
  <p>Some of the things I fail to understand here are:</p>
  <ul>
  <li>Why would a pending job kill another job, yet remain pending afterwards?</li>
  <li>How would a pending job even have privileges to kill another job in the first place?</li>
  <li>Why would this only affect '&lt;the_user&gt;'s jobs but not those of other users?</li>
  </ul>
  <p>None of this is intended to happen. I am guessing it must be caused by some settings specific to '&lt;the_user&gt;', but I cannot figure out what they are and they are not supposed to be like this. If these are settings we admins somehow caused, it was unintended.</p>
  <p><strong>Update 2</strong></p>
  <p>The problem has magically disappeared and can no longer be reproduced.</p>
  <p>NB: some details have been anonymized as <code>&lt;something&gt;</code> above.</p>