title: Slurm jobs getting starved
body: <p>So, I have an issue with my Slurm GPU queue that leads to a job starvation every now and then.</p>
  <p>Basically, I have many nodes with 1 GPU, 2 GPUs, 3 GPUs, and only 2 have 4 GPUs. The situation is as follows:</p>
  <ol>
  <li>User A submits a 4 GPU job</li>
  <li>Slurm assigns one 4 GPU node to User A's job</li>
  <li>Users B, C and D submit 1 GPU jobs and all get allocated to the second 4 GPU node</li>
  <li><strong>User E submits one 4 GPU job</strong>, it's PENDING since there are no resources to fulfill its need</li>
  <li>Users F, G, H, I...etc submit 1 GPU jobs, which get allocated to the 4 GPU node immediately as soon as any of the jobs of users B, C or D finishes</li>
  <li>More users keep submitting jobs and the 4 GPU node stays busy with these 1 GPU jobs</li>
  <li><strong>User E 4 GPU job stays waiting FOREVER</strong>, as the 4 GPUs are never available together</li>
  </ol>
  <p>Knowing that I have set the weight of 1 GPU nodes to 1, 2 GPU nodes to 2, 3 GPU nodes to 3 and 4 GPU nodes to 4, so that users priority goes to any available 1 GPU jobs, if not then 2, if not then 3, and lastly the 4.</p>
  <p>Any suggestions to eliminate or reduce starvation here (automatically)? I have jobs that wait for weeks!</p>