text: user16204: I have been submitting my snakemake pipeline to a node-shared high performance computer (HPC) via bsub the following way:
  bsub -W {resources.runtime_min} -n {threads} -M {resources.mem_mb} -R 'rusage[mem={resources.mem_mb}]' -o run_logs/{name}_%J.out -e run_logs/{name}_%J.err"
  
  Works great, no problem.
  However, I noticed that the memory limit was tasks*memory given. I am not aware of any memory issues on the HPC. The "issue" was not noticed until I asked why my memory limit was so high compared to what I gave with the bsub flag -M (I wanted to understand how everything worked).
  I was recommend to use a -hl flag to hard cap the usage. It was actually an administator that recommended it, after I asked him about how it worked. They now want to implement it for other users. I found this in the documentation too, so I understand this.
  But I see no one else is using this and it is also not suggested when you want to use the bsub outfiles to understand how much resources you need (how much memory should I use).
  Does anyone have thoughts on this?
  Somehow, it makes sense to use -hl, but I am hesitant since no one seems to use -hl for some reason, and I feel like I do not fully understand it.
  I'm asking here, because I felt I kept bugging the administrator with the same questions :'-). Thank you in advance for any advice.
  
  user3848: What this question appears to be about is capping the RAM usage on a node within a shared HPC.
  Being 'nice' on an HPC is very honourable indeed, it is however the system administrators jurisdiction. It simply requires an email to them and they will provide the authoritative answer. Thus if there were RAM usage issues within the nodes then they will be aware of it. Limiting RAM on a job is risky because the biggest 'fall-over' in bioinformatics is a job running into a RAM bottleneck. If that happens the process just freezes and literally will take 1 log more run-time. If the machine doesn't have enough RAM for the job, thats a separate issue and easily solvable via cloud computing.
  The individual who made the suggestion might be running into RAM bottleneck issues on the machine. However, they should inform the HPC administrator who could then provide an assessment whether all other users should cap their RAM usage. There are alternatives - the user experiencing RAM bottlenecks can parallelise their job and this will overcome RAM issues. This can be a bit tricky on a course-grain parallelisation with a shared HPC.
  So in summary, I wouldn't do this personally and if I was concerned I would seek the advice of the administrator who will give the definitive response. If admin says 'do it' then do it, but it your jobs take way-way longer than they should I'd relax the constraint and inform admin.
  Admin could enforce this constraint without the user base being aware of what was happening if it was a major issue on the server.
  These days if there are HPC issues its easier to shift on a cloud computer and is not worth the arguments when everyone is trying to get their jobs done before the vacation.
  Merry Christmas!
  
  From the comments ...
  
  System admin recommended this ... okay they should now inform the rest of the user base (which is what you are implying), unless there are been a specific issue. Thats a bit different and it is polite to comply with their request;
  What you can do is justify to the system admin why you job requires extra RAM;
  Systems admin might have automatically restricted the RAM access on the account - you can ask them to lift that;
  
  You said,
  
  "1 log more run-time"?
  
  That is extremely important to understand. Bioinformatic jobs can use loads of RAM, if the job exceeds the RAM capacity of the node essentially the job "freezes". What it really does is slow the time to completion of the job dramatically, so I'm saying in my experience it's about 10-fold longer to complete than normal. That is what I am calling a RAM-bottleneck.
  Final point
  On the bsub parallelisation appears to be efficiently implemented and if RAM issues are occurring parallelising the job is the best solution. This would be a work around if the systems admin wants to restrict RAM usage and as a user you are wanting quick run-time (fast completion). Not all algorithms will parallelise but most will these days.
  
original_text: 16204: <p>I have been submitting my snakemake pipeline to a node-shared high performance computer (HPC) via <a href="https://hpc.llnl.gov/banks-jobs/running-jobs/lsf-user-manual#:%7E:text=The%20bsub%20command%20is%20used,it%20with%20the%20necessary%20corrections." rel="nofollow noreferrer">bsub</a> the following way:</p>
  <pre><code>bsub -W {resources.runtime_min} -n {threads} -M {resources.mem_mb} -R 'rusage[mem={resources.mem_mb}]' -o run_logs/{name}_%J.out -e run_logs/{name}_%J.err&quot;
  </code></pre>
  <p>Works great, no problem.</p>
  <p>However, I noticed that the memory limit was <code>tasks*memory</code> given. I am not aware of any memory issues on the HPC. The &quot;issue&quot; was not noticed until I asked why my memory limit was so high compared to what I gave with the <code>bsub</code> flag <code>-M</code> (I wanted to understand how everything worked).</p>
  <p>I was recommend to use a <code>-hl</code> flag to hard cap the usage. It was actually an administator that recommended it, after I asked him about how it worked. They now want to implement it for other users. I found this in the <a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=options-hl" rel="nofollow noreferrer">documentation</a> too, so I understand this.</p>
  <p>But I see no one else is using this and it is also not suggested when you want to use the <code>bsub</code> outfiles to understand how much resources you need (<a href="https://hpc.ncsu.edu/Documents/lsf_submit_faq.php#:%7E:text=back%20to%20top-,How%20much%20memory%20does%20my%20code%20use%3F,-Do%20a%20sample" rel="nofollow noreferrer">how much memory should I use</a>).</p>
  <p>Does anyone have thoughts on this?</p>
  <p>Somehow, it makes sense to use <code>-hl</code>, but I am hesitant since no one seems to use <code>-hl</code> for some reason, and I feel like I do not fully understand it.</p>
  <p>I'm asking here, because I felt I kept bugging the administrator with the same questions :'-). Thank you in advance for any advice.</p>
   3848: <p>What this question appears to be about is capping the RAM usage on a node within a shared HPC.</p>
  <p>Being 'nice' on an HPC is very honourable indeed, it is however the system administrators jurisdiction. It simply requires an email to them and they will provide the authoritative answer. Thus if there were RAM usage issues within the nodes then they will be aware of it. Limiting RAM on a job is risky because the biggest 'fall-over' in bioinformatics is a job running into a RAM bottleneck. If that happens the process just freezes and literally will take 1 log more run-time. If the machine doesn't have enough RAM for the job, thats a separate issue and easily solvable via cloud computing.</p>
  <p>The individual who made the suggestion might be running into RAM bottleneck issues on the machine. However, they should inform the HPC administrator who could then provide an assessment whether all other users should cap their RAM usage. There are alternatives - the user experiencing RAM bottlenecks can parallelise their job and this will overcome RAM issues. This can be a bit tricky on a course-grain parallelisation with a shared HPC.</p>
  <p>So in summary, I wouldn't do this personally and if I was concerned I would seek the advice of the administrator who will give the definitive response. If admin says 'do it' then do it, but it your jobs take way-way longer than they should I'd relax the constraint and inform admin.</p>
  <p>Admin could enforce this constraint without the user base being aware of what was happening if it was a major issue on the server.</p>
  <p>These days if there are HPC issues its easier to shift on a cloud computer and is not worth the arguments when everyone is trying to get their jobs done before the vacation.</p>
  <p>Merry Christmas!</p>
  <hr />
  <p>From the comments ...</p>
  <ul>
  <li>System admin recommended this ... okay they should now inform the rest of the user base (which is what you are implying), unless there are been a specific issue. Thats a bit different and it is polite to comply with their request;</li>
  <li>What you can do is justify to the system admin why you job requires extra RAM;</li>
  <li>Systems admin might have automatically restricted the RAM access on the account - you can ask them to lift that;</li>
  </ul>
  <p>You said,</p>
  <blockquote>
  <p>&quot;1 log more run-time&quot;?</p>
  </blockquote>
  <p>That is extremely important to understand. Bioinformatic jobs can use loads of RAM, if the job exceeds the RAM capacity of the node essentially the job &quot;freezes&quot;. What it really does is slow the time to completion of the job dramatically, so I'm saying in my experience it's about 10-fold longer to complete than normal. That is what I am calling a RAM-bottleneck.</p>
  <p><strong>Final point</strong>
  On the <code>bsub</code> parallelisation appears to be efficiently implemented and if RAM issues are occurring parallelising the job is the best solution. This would be a work around if the systems admin wants to restrict RAM usage and as a user you are wanting quick run-time (fast completion). Not all algorithms will parallelise but most will these days.</p>
  