text: user136: Related to my other question (Samtools sort: most efficient memory and thread settings for many samples on a cluster), we need to optimize samtools sort as we prepare to analyze thousands of .bam files, beginning with re-alignment, sorting, etc.
  Ultimate question: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting .fastqs for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.
  Unless I'm missing something, samtools merge was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (http://www.htslib.org/doc/samtools-merge.html): "Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order."
  Is it fastest to:
  
  samtools sort each 'mini-bam' and then samtools merge?
  samtools cat the 'mini-bams' together and then samtools sort?
  Another option I haven't considered?
  
  Note: I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:
  
  Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?
  Merging BAM files before sorting
  
  
  user73: As part of its operations, samtools sort will split the input up into multiple temporary BAM files (which are each individually sorted) if the reads can't fit entirely within memory. Those files are merged for the final output. From the manpage:
  
  The  sorted output is written to standard output by default, or to the specified file (out.bam) when -o is used. This command will also create temporary files tmpprefix.%d.bam as needed when the entire alignment data cannot fit into  memory (as controlled via the -m option).
  
  Given this, I would expect that a concatenated, unsorted BAM will take more time to process than merging sorted BAM files, because in addition to creating the concatenated BAM file, the splitting would need to be done again (assuming the reads don't fit in memory) prior to the final merge carried out by samtools sort.
  I appreciate that this is not a quantified answer; it's just my guess based on what I have observed to happen when doing my own analyses.
  
original_text: 136: <p>Related to my other question (<a href="https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c">Samtools sort: most efficient memory and thread settings for many samples on a cluster</a>), we need to optimize <code>samtools sort</code> as we prepare to analyze thousands of <code>.bam</code> files, beginning with re-alignment, sorting, etc.</p>
  <p><strong>Ultimate question</strong>: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting <code>.fastq</code>s for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.</p>
  <p>Unless I'm missing something, <code>samtools merge</code> was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (<a href="http://www.htslib.org/doc/samtools-merge.html" rel="nofollow noreferrer">http://www.htslib.org/doc/samtools-merge.html</a>): &quot;Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order.&quot;</p>
  <p>Is it fastest to:</p>
  <ol>
  <li><code>samtools sort</code> each 'mini-bam' and then <code>samtools merge</code>?</li>
  <li><code>samtools cat</code> the 'mini-bams' together and then <code>samtools sort</code>?</li>
  <li>Another option I haven't considered?</li>
  </ol>
  <p><strong>Note:</strong> I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:</p>
  <ol>
  <li><a href="https://www.biostars.org/p/9864/" rel="nofollow noreferrer">Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?</a></li>
  <li><a href="http://seqanswers.com/forums/showthread.php?t=47110" rel="nofollow noreferrer">Merging BAM files before sorting</a></li>
  </ol>
   73: <p>As part of its operations, <code>samtools sort</code> will split the input up into multiple temporary BAM files (which are each individually sorted) if the reads can't fit entirely within memory. Those files are merged for the final output. From the manpage:</p>
  <blockquote>
  <p>The  sorted output is written to standard output by default, or to the specified file (out.bam) when -o is used. This command will also create temporary files tmpprefix.%d.bam as needed when the entire alignment data cannot fit into  memory (as controlled via the -m option).</p>
  </blockquote>
  <p>Given this, I would expect that a concatenated, unsorted BAM will take more time to process than merging sorted BAM files, because in addition to creating the concatenated BAM file, the splitting would need to be done again (assuming the reads don't fit in memory) prior to the final merge carried out by <code>samtools sort</code>.</p>
  <p>I appreciate that this is not a quantified answer; it's just my guess based on what I have observed to happen when doing my own analyses.</p>
  