text: user163: I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.
  I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?
  
  user134: Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency.  So samtools merge (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), picard MergeSamFiles, etc.
  These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors.  Use ulimit to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc.  Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.
  
  user77: samtools merge merged.bam *.bam is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.
  
original_text: 163: <p>I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.</p>
  
  <p>I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?</p>
   134: <p>Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency.  So <code>samtools merge</code> (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), <code>picard MergeSamFiles</code>, etc.</p>
  
  <p>These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors.  Use <code>ulimit</code> to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc.  Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.</p>
   77: <p><code>samtools merge merged.bam *.bam</code> is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.</p>
  