text: user146: I am constructing a bit of software which pipes the outputs of the bam file via samtools view into a script for parsing. My goal is to (somehow) make this process more efficient, and faster than samtools view. 
  I am only using 3-4 fields in the bam. Naturally, if I only wanted those fields, I could do something like samtools view file1.bam | cut -f ##. 
  However, I'm trying to be faster than this approach. More specifically: 
  (1) Given the bam file format, is it theoretically possible to output several requested SAM fields faster than samtools view|cut -f? 
  (2) Is there software available to achieve (1), or would one need to hack samtools? 
  
  user272: Another library you can use for this purpose is the htsjdk, which is written in java. Using htsjdk with java is analogous to using htslib with C; the BAM format is already handled by the library and you can manipulate fields in your own code. The same basic analysis applies to java as C; you do not need to convert the BAM file to text and then parse it. Generally, a program implemented in java will be slower than C but still significantly faster than scripting. You do get language features of java: portability, array bounds checking, garbage collection, etc. 
  
  user73: The BAM file format is not a text-based format. It has a specific binary structure, specified in reasonable detail in the SAM file format specification. Whenever this information is displayed on a screen as text, it needs to be converted from the binary format to a text format, which takes a bit of time and processing power.
  As this question suggests, if only a specific field from alignments is needed (or fields), then it will probably be better to extract just those fields and do any necessary conversion only on those fields. While this can be done by writing a BAM parser from scratch, many developers have already written software libraries to process BAM files in this way.
  Devon Ryan has suggested htslib, which is a C library written by a group that includes the people who wrote the SAM/BAM/CRAM file format specifications. There's also pysam, which is a python wrapper around htslib.
  The particular tool that is used will depend on your familiarity with programming and the specific thing that you want to do. If you just want to "output a BAM file", then cat is the quickest:
  cat file1.bam
  
  ... but you probably don't want to do that, because it seems like you want to process a text-based representation with a script. Because you haven't specified which fields you are interested in, it's not possible to suggest the best thing to use. In the end, I expect that an "efficient" solution to your problem would involve htslib in some form. This is not really hacking samtools, it's using the backend of samtools to process BAM data.
  However, it's useful to also think about the time cost of coding. What application do you have which means that the text-based processing method is not sufficient? It takes time to write code, and a lot more time to debug that code to make sure it's doing the right thing in all situations. If this is for a one-off thing, then samtools view output fed into your script may be the quickest solution.
  
  user37: I modified your original question: as you are extracting 4 fields, you are not outputting BAM. The answer to the modified question is: yes, you can write a C program with htslib (or with bamtools, bioD, bioGo or rust-bio). Formatting an entire SAM is fairly expensive. You can see this by comparing samtools view aln.bam > /dev/null and samtools view -u aln.bam > /dev/null. With a C program, you can select fields to output. This will give you a noticeable performance boost, depending on the fields you extract.
  In general, if you really care about performance, don't use script. The time you save from optimizing samtools view|cut -f will be quickly neutralized by the inefficiency of most scripting languages.
  
original_text: 146: <p>I am constructing a bit of software which pipes the outputs of the bam file via <code>samtools view</code> into a script for parsing. My goal is to (somehow) make this process more efficient, and faster than <code>samtools view</code>. </p>
  
  <p>I am only using 3-4 fields in the bam. Naturally, if I only wanted those fields, I could do something like <code>samtools view file1.bam | cut -f ##</code>. </p>
  
  <p>However, I'm trying to be faster than this approach. More specifically: </p>
  
  <p>(1) Given the bam file format, is it theoretically possible to output several requested SAM fields faster than <code>samtools view|cut -f</code>? </p>
  
  <p>(2) Is there software available to achieve (1), or would one need to hack samtools? </p>
   272: <p>Another library you can use for this purpose is the <a href="https://github.com/samtools/htsjdk" rel="nofollow noreferrer">htsjdk</a>, which is written in java. Using htsjdk with java is analogous to using htslib with C; the BAM format is already handled by the library and you can manipulate fields in your own code. The same basic analysis applies to java as C; you do not need to convert the BAM file to text and then parse it. Generally, a program implemented in java will be slower than C but still significantly faster than scripting. You do get language features of java: portability, array bounds checking, garbage collection, etc. </p>
   73: <p>The BAM file format is not a text-based format. It has a specific binary structure, specified in reasonable detail in the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="noreferrer">SAM file format specification</a>. Whenever this information is displayed on a screen as text, it needs to be converted from the binary format to a text format, which takes a bit of time and processing power.</p>
  
  <p>As this question suggests, if only a specific field from alignments is needed (or fields), then it will probably be better to extract just those fields and do any necessary conversion only on those fields. While this <em>can</em> be done by writing a BAM parser from scratch, many developers have already written software libraries to process BAM files in this way.</p>
  
  <p>Devon Ryan has suggested <a href="http://www.htslib.org/" rel="noreferrer">htslib</a>, which is a C library written by a group that includes the people who wrote the SAM/BAM/CRAM file format specifications. There's also <a href="https://github.com/pysam-developers/pysam" rel="noreferrer">pysam</a>, which is a python wrapper around htslib.</p>
  
  <p>The particular tool that is used will depend on your familiarity with programming and the specific thing that you want to do. If you just want to "output a BAM file", then cat is the quickest:</p>
  
  <pre><code>cat file1.bam
  </code></pre>
  
  <p>... but you probably don't want to do that, because it seems like you want to process a text-based representation with a script. Because you haven't specified which fields you are interested in, it's not possible to suggest the best thing to use. In the end, I expect that an "efficient" solution to your problem would involve <code>htslib</code> in some form. This is not really hacking <em>samtools</em>, it's using the backend of samtools to process BAM data.</p>
  
  <p>However, it's useful to also think about the time cost of coding. What application do you have which means that the text-based processing method is not sufficient? It takes time to write code, and a lot more time to debug that code to make sure it's doing the right thing in all situations. If this is for a one-off thing, then <code>samtools view</code> output fed into your script may be the quickest solution.</p>
   37: <p>I modified your original question: as you are extracting 4 fields, you are not outputting BAM. The answer to the modified question is: yes, you can write a C program with htslib (or with bamtools, bioD, bioGo or rust-bio). Formatting an entire SAM is fairly expensive. You can see this by comparing <code>samtools view aln.bam &gt; /dev/null</code> and <code>samtools view -u aln.bam &gt; /dev/null</code>. With a C program, you can select fields to output. This will give you a noticeable performance boost, depending on the fields you extract.</p>
  
  <p>In general, if you really care about performance, don't use script. The time you save from optimizing <code>samtools view|cut -f</code> will be quickly neutralized by the inefficiency of most scripting languages.</p>
  