text: user57: I have three sequencing libraries of single individual mapped to a reference using bwa-mem. I would like to merge the three unsorted .sam files I have so, I can call variants and heterozygosity estimates using atlas. Atlas requires one input mapping file (bam) with defined read groups because it estimates the error profiles of different libraries separately.
  How can I merge multiple sam files? Preferably avoiding java (Picard tools).
  I tried to figure out a solution using samtools 1.3. I sorted individual files using samtools sort, then I used samtools merge -r merged.bam s1.sort.sam s2.sort.sam s3.sort.sam to merge the sorted files. However, the read group didn't make it to the header (and the variant caller I use is complaining about it), also the read group is stupidly the file name.
  I tried to define meaningful readgroup names using procedure described on BioStars, but I found that this will just change the header, it does not adjust the names of read groups defined by samtools merge (the file names).
  Following this related thread on SeqAnswers, I tried to define the correct header with read groups corresponding to merged file names:
  samtools -rh rg.txt merged.bam s1.sort.sam s2.sort.sam s3.sort.sam
  
  where rg.txt is a file with header
  @RG s1.sort
  @RG s2.sort
  @RG s3.sort
  ...
  output of samtools view -H s1.sort
  
  However, the header still had not the read group, I guess because sam header accepts only tagged items to be specified (something like @RG    XY:s1.sort). So, I looked into the merged bam and I found that the tag of RG is Z:. So I tried to just rename header of the merged file using samtools reheader, but then samtools complain about the fact that a tag needs to be of length 2:
  Malformed key:value pair at line 123: "@RG    Z:s1.sort"
  Segmentation fault (core dumped)
  
  I have opened an issue to report this strange incompatibility of read groups generated by samtools merge with samtools reheader.
  I would like solution to :
  
  create a bit more standardized read group names (SM:Sample\tLB:library format)
  avoid pointless writing to disk like in sam -> sorted.sam -> merged.bam case (can be probably achieved through "pipes and tees", thanks @bli)
  
  I also know that I can specify RG to bwa, so the sam files will have read groups defined in the first place. But I do not like the idea of remapping three libraries just to create correct formatting of read groups.
  
  user57: Not so elegant but working solution
  I found a solution that satisfy several of my conditions, basically I just have to assign read groups to individual mapping files, which can be just added to sorting process and then merge the sorted sam files with read groups.
  function sort_and_assign_RG {
      # $1 input file
      # $2 read group id
      # $3 library
  
      OFILE=$(basename $1 .sam).sort.RG.sam
      HEADER=$(basename $1 .sam).header.sam
  
      # since I take header from unsorted file, so I need to add this information to header as well as the readgroup
      echo -e "@HD\tVN:1.3\tSO:coordinate" > $HEADER
      # add read group to header
      echo -e "@RG\tID:"$2"\tLB:""$3" >> $HEADER
      # and the rest of the header
      samtools view -H $1 >> $HEADER
  
      # now sort the input sam, remove header, 
      # attach to each alignment read group ID (RGID) 
      # and cat header and alignment together
      samtools sort -- $1 | samtools view - | \
        awk -v RGID="$2" '{ printf "%s\tRG:Z:%s\n", $0, RGID; }' | \
        cat $HEADER - > $OFILE
  
      # remove temp header
      rm $HEADER
  }
  
  sort_and_assign_RG s1.sam s1 is180
  sort_and_assign_RG s2.sam s2 is350
  sort_and_assign_RG s3.sam s3 is550
  
  samtools merge merged.bam s{1,2,3}.sort.RG.sam
  
  It would be nice to avoid creation of three sam files on the way, but at least I get standardized read groups in a merged bam tile.
  --- edit ---
  I also found where my confusion is coming from. The read group in header is in format 
  @RG    ID:foo    LB:lib_foo    ...`
  @RG    ID:bar    LB:lib_bar    ...`
  
  where ID is read group ID (not sample ID as I thought) and other tags are just specifications of the read group. Then individual mapping have read group assigned in different format
  {aliment1}    RG:Z:foo
  {aliment2}    RG:Z:bar
  
  where RG is just a tag for read group and Z is just a mark that says that this tag is just a "printable string". Therefore I think that another solution would just to merge these sorted files and then just adding three correct lines to the header.
  I have to say, .sam is a unbelievable intuitive format.
  
  user964: I have used https://github.com/ekg/bamaddrg to add read groups quickly to multiple sam files.  And then you can do a samtools merge of the tagged files.
  
original_text: 57: <p>I have three sequencing libraries of single individual mapped to a reference using <code>bwa-mem</code>. I would like to merge the three unsorted <code>.sam</code> files I have so, I can call variants and heterozygosity estimates using <a href="https://bitbucket.org/phaentu/atlas/wiki/Home" rel="nofollow noreferrer">atlas</a>. Atlas requires one input mapping file (<code>bam</code>) with defined read groups because it estimates the error profiles of different libraries separately.</p>
  
  <p>How can I merge multiple sam files? Preferably avoiding java (Picard tools).</p>
  
  <p>I tried to figure out a solution using <code>samtools 1.3</code>. I sorted individual files using <code>samtools sort</code>, then I used <code>samtools merge -r merged.bam s1.sort.sam s2.sort.sam s3.sort.sam</code> to merge the sorted files. However, the read group didn't make it to the header (and the variant caller I use is complaining about it), also the read group is stupidly the file name.</p>
  
  <p>I tried to define meaningful readgroup names using procedure described on <a href="https://www.biostars.org/p/47487/" rel="nofollow noreferrer">BioStars</a>, but I found that this will just change the header, it does not adjust the names of read groups defined by <code>samtools merge</code> (the file names).</p>
  
  <p>Following this related thread on <a href="http://seqanswers.com/forums/showthread.php?t=4180" rel="nofollow noreferrer">SeqAnswers</a>, I tried to define the correct header with read groups corresponding to merged file names:</p>
  
  <pre><code>samtools -rh rg.txt merged.bam s1.sort.sam s2.sort.sam s3.sort.sam
  </code></pre>
  
  <p>where <code>rg.txt</code> is a file with header</p>
  
  <pre><code>@RG s1.sort
  @RG s2.sort
  @RG s3.sort
  ...
  output of samtools view -H s1.sort
  </code></pre>
  
  <p>However, the header still had not the read group, I guess because sam header accepts only tagged items to be specified (something like <code>@RG    XY:s1.sort</code>). So, I looked into the merged <code>bam</code> and I found that the tag of RG is <code>Z:</code>. So I tried to just rename header of the merged file using <code>samtools reheader</code>, but then samtools complain about the fact that a tag needs to be of length 2:</p>
  
  <pre><code>Malformed key:value pair at line 123: "@RG    Z:s1.sort"
  Segmentation fault (core dumped)
  </code></pre>
  
  <p>I have opened an <a href="https://github.com/samtools/samtools/issues/698" rel="nofollow noreferrer">issue</a> to report this strange incompatibility of read groups generated by <code>samtools merge</code> with <code>samtools reheader</code>.</p>
  
  <p>I would like solution to :</p>
  
  <ul>
  <li>create a bit more standardized read group names (<code>SM:Sample\tLB:library</code> format)</li>
  <li>avoid pointless writing to disk like in sam -> sorted.sam -> merged.bam case (can be probably achieved through "pipes and tees", thanks @bli)</li>
  </ul>
  
  <p>I also know that I can specify RG to bwa, so the sam files will have read groups defined in the first place. But I do not like the idea of remapping three libraries just to create correct formatting of read groups.</p>
   57: <p><strong>Not so elegant but working solution</strong></p>
  
  <p>I found a solution that satisfy several of my conditions, basically I just have to assign read groups to individual mapping files, which can be just added to sorting process and then merge the sorted sam files with read groups.</p>
  
  <pre class="lang-js prettyprint-override"><code>function sort_and_assign_RG {
      # $1 input file
      # $2 read group id
      # $3 library
  
      OFILE=$(basename $1 .sam).sort.RG.sam
      HEADER=$(basename $1 .sam).header.sam
  
      # since I take header from unsorted file, so I need to add this information to header as well as the readgroup
      echo -e "@HD\tVN:1.3\tSO:coordinate" &gt; $HEADER
      # add read group to header
      echo -e "@RG\tID:"$2"\tLB:""$3" &gt;&gt; $HEADER
      # and the rest of the header
      samtools view -H $1 &gt;&gt; $HEADER
  
      # now sort the input sam, remove header, 
      # attach to each alignment read group ID (RGID) 
      # and cat header and alignment together
      samtools sort -- $1 | samtools view - | \
        awk -v RGID="$2" '{ printf "%s\tRG:Z:%s\n", $0, RGID; }' | \
        cat $HEADER - &gt; $OFILE
  
      # remove temp header
      rm $HEADER
  }
  
  sort_and_assign_RG s1.sam s1 is180
  sort_and_assign_RG s2.sam s2 is350
  sort_and_assign_RG s3.sam s3 is550
  
  samtools merge merged.bam s{1,2,3}.sort.RG.sam
  </code></pre>
  
  <p>It would be nice to avoid creation of three <code>sam</code> files on the way, but at least I get standardized read groups in a merged bam tile.</p>
  
  <p><em>--- edit ---</em></p>
  
  <p>I also found where my confusion is coming from. The read group in header is in format </p>
  
  <pre><code>@RG    ID:foo    LB:lib_foo    ...`
  @RG    ID:bar    LB:lib_bar    ...`
  </code></pre>
  
  <p>where <code>ID</code> is read group ID (not sample ID as I thought) and other tags are just specifications of the read group. Then individual mapping have read group assigned in different format</p>
  
  <pre><code>{aliment1}    RG:Z:foo
  {aliment2}    RG:Z:bar
  </code></pre>
  
  <p>where <code>RG</code> is just a tag for read group and <code>Z</code> is just a mark that says that this tag is just a "printable string". Therefore I think that another solution would just to merge these sorted files and then just adding three <strong>correct</strong> lines to the header.</p>
  
  <p>I have to say, <code>.sam</code> is a unbelievable intuitive format.</p>
   964: <p>I have used <a href="https://github.com/ekg/bamaddrg" rel="noreferrer">https://github.com/ekg/bamaddrg</a> to add read groups quickly to multiple sam files.  And then you can do a samtools merge of the tagged files.</p>
  