text: user136: We're preparing to analyze thousands of .bam files beginning with re-alignment, sorting, etc.
  Complicated question: Has anyone investigated the optimal thread and memory settings to sort hundreds to thousands of .bam files using samtools sort on a cluster?
  By 'optimal', I'm looking to sort as many samples as possible in the shortest time frame. I'm mostly interested in optimizing samtools sort because the other steps (e.g., alignment) are much more obvious to fine-tune.
  For a single sample, throwing large amounts of memory + threads is trivial, but that isn't scalable for thousands of samples. There is also likely a diminishing return at a certain point, even for a single sample.
  Ideally, I'd like to see plots identifying the 'sweet spot' for threads + memory for a single sample (e.g., x == threads, y == memory, z == time), and then plots maximizing throughput for more samples. I would expect the data to follow a -log function, where we could identify the optimal settings at the knee in the curve.
  There are obviously many other factors (e.g., cluster size, administrative scheduler settings, etc.), but I suspect findings on a given cluster will generalize to other clusters.
  Really appreciate any feedback.
  
  user136: Update 2
  Ok, for what I expect will be my final update for this answer, I compared the following:
  
  samtools 1.14 + zlib
  samtools 1.15 + libdeflate (this is a different version, but shouldn't have a major effect. They released a new version just before I asked cluster admins to compile against libdeflate and didn't notice until later.)
  sambamba v0.8.2
  
  Note: This time I used the following settings:
  
  Only two samples instead of three
  Max Nextflow queue size of 30 to avoid too many threads reading from the same two files.
  I tested CPU requests from 1 to 7 (step size 1) and then from 9 to 17 (step size 2).
  For jobs where I allotted few CPUs and low memory, I provided a 200% buffer to prevent the jobs from failing with OUT OF MEMORY errors. i.e., I gave the SLURM job 2x the amount of memory than I told the tools they could use. sambamba was the worst offender for exceeding memory in these conditions, but I gave them all 200% buffer to keep comparisons equal.
  
  Seems pretty clear to me that sambamba is the fastest and uses requested CPU resources most efficiently (i.e., is able to use all of the CPUs requested up until ~8 or 9 CPUs). For reference, the difference between sambamba and samtools + libdeflate with 9 CPUs is ~10 minutes on average.
  We will certainly be using sambamba going forward, probably with ~9 CPUs and ~9GB per thread since I had to provide a 200% buffer in lower CPU/Mem jobs to avoid job failure, anyway.
  Again, all code and results are available on my GitHub repository (Samtools sort optimization test).
  (Angle 1) Comparison between samtools + zlib, samtools + libdeflate (Lsamtools), and sambamba
  Here, I'm showing the general trend that real time decreases with increased CPUs for all three tools, but Mem Per CPU doesn't have a huge effect. sambamba out performs samtools + zlib regardless of settings. samtools + libdeflate out performs samtools + zlib until ~11 CPUs, where they perform the same (plot 3 below explains why).
  Interesting finding: samtools + libdeflate performs better than sambamba with a single thread, but sambamba takes over from there as CPUs increase (again, plot #3 explains why).
  
  (Angle 2) Comparison between samtools + zlib, samtools + libdeflate (Lsamtools), and sambamba
  Here, I'm showing that samtools + zlib and sambamba performance flattens out at ~11 CPUs (again, plot 3 below explans why).
  
  Requested CPUs vs. CPU utilization for samtools + zlib, samtools + libdeflate (Lsamtools), and sambamba
  Here, I'm showing how well each tool is able to utilize the CPUs allotted to it. It was fascinating to see that samtools + libdeflate flattens out very early in it's CPU utilization (i.e., ~3 CPUS) and struggles to cross the 400% CPU utilization with even >10 CPUs. This is why samtools + libdeflate performance flattens out quickly.
  samtools + zlib does a much better job utilizing allotted CPUs, but still lags behind sambamba throughout (and is slower, anyway).
  
  Update 1
  
  My first plots (below) were showing duration on the z-axis, which is the time from job submission to completion rather than job start to completion. That's why the larger memory requests were much longer in the original plots. Duration is still a useful measure, but I'm now including real time, which is from start to completion.
  Judging solely by real time (above), there is a clear gain between 5 & 7 CPUs (~10 minutes, on average when compared to the same mem-per-thread).
  More data coming, including samtools compiled against libdeflate and sambamba.
  
  Original post
  I made time to semi-formally look at this. I'm glad I did because the results surprised me. I don't answer all of my questions directly, but these results are good enough to get me going in the right direction.
  My code and results are in my GitHub repository (Samtools_sort_optimization_test) where you will also find an interactive plot you can download to see the results more clearly (see SAMTOOLS_SORT/samtools_sort_CPU_and_memory_comparison.tar.gz in the repository). It's hard to truly appreciate the 3D plotly plot until you start rotating it.
  NOTES:
  
  Even though this is from only three samples, I think my results are accurate. I welcome any feedback in case there's a bug.
  In my experience, samtools often exceeds its allotted memory (at least on a SLURM scheduler). i.e., the job gets killed by SLURM. To avoid this, I requested an additional 20% of memory to provide a buffer for samtools. I don't know if this discrepancy is because of SLURM or samtools, though.
  These results are using samtools compiled against zlib. We are preparing a version of samtools compiled against libdeflate, per @user172818's suggestion. I will add those results once completed. I may also compare against sambamba, per @acvill's suggestion.
  These results are related to my other question about the best sorting/merging method. I will post those results in that question and provide a brief summary here.
  
  tl;dr (too long, didn't read)
  Primary conclusions based on the plots below
  
  Optimal CPUs (results as expected): There's a significant gain with some additional CPUs, but it levels off around 7 CPUs (total, not 'additional'; i.e., samtools sort -@ 6). I thought we might still see some nice gains above 7, but this is probably what most would expect (e.g., similar to what @user172818 suggested).
  Optimal memory per thread (results NOT as expected): There is very little to gain with more than 1GB of memory (per thread); that surprised me. There are some performance gains up until ~7GB per thread (~2-4 minutes faster). What really surprised me, however, is that performance dramatically decreases as memory per thread goes >7GB. The updated plot (above) shows there is some gain with >1GB mem per thread. When running lots of samples at once (hundreds to 1000), mem per thread seems to play a much larger role (data coming).
  Effect from sorting status (results as expected): Largely in an attempt to do this as quickly as possible (laziness?), I used three samples, where one was from an aligned and unsorted .bam file, and the other two were from .bam files that had been aligned and sorted, but I then collated them (samtools collate), rather than re-aligning them. Unsurprisingly, the collated samples (not the average use case) sorted slightly faster than the
  unsorted sample (average use case).
  
  Initial opinion
  
  "Small" sample set: For a small sample set, I would probably use 7 total CPUs and 7GB per thread.
  "Large" sample set: I will probably use 7 CPUs (maybe 5) with only 1GB memory per thread. That will allow the scheduler to keep as many jobs going as possible.
  
  samtools sort: duration vs. CPUs and mem per thread
  As mentioned above, an interactive version of this plot is available on my GitHub.
  Angle 1
  
  Angle 2
  
  Additional details
  Analyses were run on University of Kentucky's MCC Cluster, which uses AMD 7702P processors, using samtools v.1.14
  
  user37: I usually use samtools sort -@4 -m4g. Samtools will be faster with more threads and more RAM, but I feel the gain is not significant. I haven't done a proper evaluation, but I have run samtools sort with different settings from time to time on clusters when I have to consider speed-resource tradeoffs.
  When you use samtools, make sure you compile it with libdeflate, which is about twice as fast as zlib on compression. I don't know if the bioconda version built with libdeflate. You may consider the precompiled binary here.
  Samtools sort, especially with libdeflate, is fairly fast these days. Performance metrics in the other answer are outdated.
  
original_text: 136: <p>We're preparing to analyze thousands of <code>.bam</code> files beginning with re-alignment, sorting, etc.</p>
  <p><strong>Complicated question:</strong> Has anyone investigated the optimal thread and memory settings to sort hundreds to thousands of <code>.bam</code> files using <code>samtools sort</code> on a cluster?</p>
  <p>By 'optimal', I'm looking to sort as many samples as possible in the shortest time frame. I'm mostly interested in optimizing <code>samtools sort</code> because the other steps (e.g., alignment) are much more obvious to fine-tune.</p>
  <p>For a single sample, throwing large amounts of memory + threads is trivial, but that isn't scalable for thousands of samples. There is also likely a <a href="https://en.wikipedia.org/wiki/Diminishing_returns" rel="nofollow noreferrer">diminishing return</a> at a certain point, even for a single sample.</p>
  <p>Ideally, I'd like to see plots identifying the 'sweet spot' for threads + memory for a single sample (e.g., x == threads, y == memory, z == time), and then plots maximizing throughput for more samples. I would expect the data to follow a -log function, where we could identify the optimal settings at the <a href="https://en.wikipedia.org/wiki/Knee_of_a_curve#:%7E:text=In%20mathematics%2C%20a%20knee%20of,or%20in%20the%20other%20direction." rel="nofollow noreferrer">knee in the curve</a>.</p>
  <p>There are obviously many other factors (e.g., cluster size, administrative scheduler settings, etc.), but I suspect findings on a given cluster will generalize to other clusters.</p>
  <p>Really appreciate any feedback.</p>
   136: <h1>Update 2</h1>
  <p>Ok, for what I expect will be my final update for this answer, I compared the following:</p>
  <ol>
  <li><code>samtools 1.14</code> + <code>zlib</code></li>
  <li><code>samtools 1.15</code> + <code>libdeflate</code> (this is a different version, but shouldn't have a major effect. They released a new version just before I asked cluster admins to compile against <code>libdeflate</code> and didn't notice until later.)</li>
  <li><code>sambamba v0.8.2</code></li>
  </ol>
  <p><strong>Note</strong>: This time I used the following settings:</p>
  <ol>
  <li>Only two samples instead of three</li>
  <li>Max <code>Nextflow</code> queue size of 30 to avoid too many threads reading from the same two files.</li>
  <li>I tested CPU requests from 1 to 7 (step size 1) and then from 9 to 17 (step size 2).</li>
  <li>For jobs where I allotted few CPUs and low memory, I provided a 200% buffer to prevent the jobs from failing with <code>OUT OF MEMORY</code> errors. i.e., I gave the <code>SLURM</code> job 2x the amount of memory than I told the tools they could use. <code>sambamba</code> was the worst offender for exceeding memory in these conditions, but I gave them all 200% buffer to keep comparisons equal.</li>
  </ol>
  <p>Seems pretty clear to me that <code>sambamba</code> is the fastest and uses requested CPU resources most efficiently (i.e., is able to use all of the CPUs requested up until ~8 or 9 CPUs). For reference, <strong>the difference between <code>sambamba</code> and <code>samtools + libdeflate</code> with 9 CPUs is ~10 minutes on average.</strong></p>
  <p>We will certainly be using <code>sambamba</code> going forward, probably with ~9 CPUs and ~9GB per thread since I had to provide a 200% buffer in lower CPU/Mem jobs to avoid job failure, anyway.</p>
  <p>Again, all code and results are available on my GitHub repository (<a href="https://github.com/mebbert/Samtools_sort_optimization_test" rel="nofollow noreferrer">Samtools sort optimization test</a>).</p>
  <h2>(Angle 1) Comparison between <code>samtools + zlib</code>, <code>samtools + libdeflate</code> (Lsamtools), and <code>sambamba</code></h2>
  <p>Here, I'm showing the general trend that <code>real time</code> decreases with increased CPUs for all three tools, but Mem Per CPU doesn't have a huge effect. <code>sambamba</code> out performs <code>samtools + zlib</code> regardless of settings. <code>samtools + libdeflate</code> out performs <code>samtools + zlib</code> until ~11 CPUs, where they perform the same (plot 3 below explains why).</p>
  <p><strong>Interesting finding:</strong> <code>samtools + libdeflate</code> performs better than <code>sambamba</code> with a single thread, but <code>sambamba</code> takes over from there as CPUs increase (again, plot #3 explains why).</p>
  <p><a href="https://i.stack.imgur.com/87BoN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/87BoN.png" alt="samtools, Lsamtools, and sambamba" /></a></p>
  <h2>(Angle 2) Comparison between <code>samtools + zlib</code>, <code>samtools + libdeflate</code> (Lsamtools), and <code>sambamba</code></h2>
  <p>Here, I'm showing that <code>samtools + zlib</code> and <code>sambamba</code> performance flattens out at ~11 CPUs (again, plot 3 below explans why).</p>
  <p><a href="https://i.stack.imgur.com/B8WfV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/B8WfV.png" alt="samtools, Lsamtools, and sambamba" /></a></p>
  <h2>Requested CPUs vs. CPU utilization for <code>samtools + zlib</code>, <code>samtools + libdeflate</code> (Lsamtools), and <code>sambamba</code></h2>
  <p>Here, I'm showing how well each tool is able to utilize the CPUs allotted to it. It was fascinating to see that <code>samtools + libdeflate</code> flattens out very early in it's CPU utilization (i.e., ~3 CPUS) and struggles to cross the 400% CPU utilization with even &gt;10 CPUs. This is why <code>samtools + libdeflate</code> performance flattens out quickly.</p>
  <p><code>samtools + zlib</code> does a much better job utilizing allotted CPUs, but still lags behind <code>sambamba</code> throughout (and is slower, anyway).
  <a href="https://i.stack.imgur.com/eiSsn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eiSsn.png" alt="CPU usage by requested CPUs" /></a></p>
  <h1>Update 1</h1>
  <p><a href="https://i.stack.imgur.com/jTNaG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jTNaG.png" alt="Realtime vs. CPUs and mem per thread (samtools sort)" /></a></p>
  <p>My first plots (below) were showing <code>duration</code> on the <code>z-axis</code>, which is the time from job <em>submission</em> to <em>completion</em> rather than job <em>start</em> to <em>completion</em>. That's why the larger memory requests were much longer in the original plots. <code>Duration</code> is still a useful measure, but I'm now including <code>real time</code>, which is from <em>start</em> to <em>completion</em>.</p>
  <p>Judging solely by <code>real time</code> (above), there is a clear gain between 5 &amp; 7 CPUs (~10 minutes, on average when compared to the same mem-per-thread).</p>
  <p>More data coming, including <code>samtools</code> compiled against <code>libdeflate</code> and <code>sambamba</code>.</p>
  <hr />
  <h1>Original post</h1>
  <p>I made time to semi-formally look at this. I'm glad I did because the results surprised me. I don't answer all of my questions directly, but these results are good enough to get me going in the right direction.</p>
  <p>My code and results are in my <code>GitHub</code> repository (<a href="https://github.com/mebbert/Samtools_sort_optimization_test" rel="nofollow noreferrer">Samtools_sort_optimization_test</a>) where you will also find an interactive plot you can download to see the results more clearly (see <code>SAMTOOLS_SORT/samtools_sort_CPU_and_memory_comparison.tar.gz</code> in the repository). It's hard to truly appreciate the 3D <code>plotly</code> plot until you start rotating it.</p>
  <p><strong>NOTES:</strong></p>
  <ol>
  <li>Even though this is from only three samples, I think my results are accurate. I welcome any feedback in case there's a bug.</li>
  <li>In my experience, <code>samtools</code> often exceeds its allotted memory (at least on a <code>SLURM</code> scheduler). i.e., the job gets killed by <code>SLURM</code>. To avoid this, I requested an additional 20% of memory to provide a buffer for <code>samtools</code>. I don't know if this discrepancy is because of <code>SLURM</code> or <code>samtools</code>, though.</li>
  <li>These results are using <code>samtools</code> compiled against <code>zlib</code>. We are preparing a version of <code>samtools</code> compiled against <code>libdeflate</code>, per @user172818's suggestion. I will add those results once completed. I may also compare against <code>sambamba</code>, per @acvill's suggestion.</li>
  <li>These results are related to my other question about <a href="https://bioinformatics.stackexchange.com/questions/18539/samtools-sort-most-efficient-approach-to-sort-a-single-sample-after-aligning-sp">the best sorting/merging method</a>. I will post those results in that question and provide a brief summary here.</li>
  </ol>
  <h1>tl;dr (too long, didn't read)</h1>
  <h2>Primary conclusions based on the plots below</h2>
  <ol>
  <li><strong>Optimal CPUs</strong> (results as expected): There's a significant gain with some additional CPUs, but it levels off around 7 CPUs (total, not 'additional'; i.e., <code>samtools sort -@ 6</code>). I thought we might still see <em>some</em> nice gains above 7, but this is probably what most would expect (e.g., similar to what @user172818 suggested).</li>
  <li><strong>Optimal memory per thread</strong> (<s>results NOT as expected</s>): <s>There is very little to gain with more than 1GB of memory (per thread); that surprised me. There are <em>some</em> performance gains up until ~7GB per thread (~2-4 minutes faster). What really surprised me, however, is that performance dramatically <strong>decreases</strong> as memory per thread goes &gt;7GB</s>. The updated plot (above) shows there is some gain with &gt;1GB mem per thread. When running lots of samples at once (hundreds to 1000), mem per thread seems to play a much larger role (data coming).</li>
  <li><strong>Effect from sorting status</strong> (results as expected): Largely in an attempt to do this as quickly as possible (laziness?), I used three samples, where one was from an aligned and unsorted <code>.bam</code> file, and the other two were from <code>.bam</code> files that had been aligned and sorted, but I then collated them (<code>samtools collate</code>), rather than re-aligning them. Unsurprisingly, the collated samples (not the average use case) sorted slightly faster than the
  unsorted sample (average use case).</li>
  </ol>
  <h2>Initial opinion</h2>
  <ol>
  <li><strong>&quot;Small&quot; sample set</strong>: For a small sample set, I would probably use 7 total CPUs and 7GB per thread.</li>
  <li><strong>&quot;Large&quot; sample set</strong>: I will probably use 7 CPUs (maybe 5) with only 1GB memory per thread. That will allow the scheduler to keep as many jobs going as possible.</li>
  </ol>
  <h2><code>samtools sort</code>: duration vs. CPUs and mem per thread</h2>
  <p>As mentioned above, an interactive version of this plot is available on my <code>GitHub</code>.</p>
  <p><strong>Angle 1</strong>
  <a href="https://i.stack.imgur.com/o79Ek.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/o79Ek.png" alt="Duration vs. CPUs and mem per thread (samtools sort)" /></a></p>
  <p><strong>Angle 2</strong>
  <a href="https://i.stack.imgur.com/UGNSu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UGNSu.png" alt="Duration vs. CPUs and mem per thread (samtools sort)" /></a></p>
  <h1>Additional details</h1>
  <p>Analyses were run on <a href="https://docs.ccs.uky.edu/display/HPC/MCC+system+overview" rel="nofollow noreferrer">University of Kentucky's MCC Cluster</a>, which uses <code>AMD 7702P</code> processors, using <code>samtools v.1.14</code></p>
   37: <p>I usually use <code>samtools sort -@4 -m4g</code>. Samtools will be faster with more threads and more RAM, but I <em>feel</em> the gain is not significant. I haven't done a proper evaluation, but I have run samtools sort with different settings from time to time on clusters when I have to consider speed-resource tradeoffs.</p>
  <p>When you use samtools, make sure you compile it with <a href="https://github.com/ebiggers/libdeflate" rel="noreferrer">libdeflate</a>, which is about twice as fast as zlib on compression. I don't know if the bioconda version built with libdeflate. You may consider the precompiled binary <a href="https://zenodo.org/record/5731013" rel="noreferrer">here</a>.</p>
  <p>Samtools sort, especially with libdeflate, is fairly fast these days. Performance metrics in the other answer are outdated.</p>
  