text: user43: I'd like to learn the differences between 3 common formats such as FASTA, FASTQ and SAM. How they are different? Are there any benefits of using one over another?
  Based on Wikipedia pages, I can't tell the differences between them.
  
  user94: FASTA and FASTQ formats are both file formats that contain sequencing reads while SAM files are these reads aligned to a reference sequence. In other words, FASTA and FASTQ are the "raw data" of sequencing while SAM is the product of aligning the sequencing reads to a refseq.
  A FASTA file contains a read name followed by the sequence. An example of one of these reads for RNASeq might be:
  >Flow cell number: lane number: chip coordinates etc.
  ATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTA
  
  The FASTQ version of this read will have two more lines, one + as a place holder and then a line of quality scores for the base calls. The qualities are given as characters with '!' being the lowest and '~' being the highest, in increasing ASCII value. It would look something like this
  @Flow cell number: lane number: chip coordinates etc.
  ATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTA
  +
  !''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65
  
  A SAM file has many fields for each alignment, the header begins with the @ character. The alignment contains 11 mandatory fields and various optional ones. You can find the spec file here: https://samtools.github.io/hts-specs/SAMv1.pdf .
  Often you'll see BAM files which are just compressed binary versions of SAM files. You can view these alignment files using various tools, such as SAMtools, IGV or USCS Genome browser.
  As to the benefits, FASTA/FASTQ vs. SAM/BAM is comparing apples and oranges. I do a lot of RNASeq work so generally we take the FASTQ files and align them the a refseq using an aligner such as STAR which outputs SAM/BAM files. There's a lot you can do with just these alignment files, looking at expression, but usually I'll use a tool such as RSEM to "count" the reads from various genes to create an expression matrix, samples as columns and genes as rows. Whether you get FASTQ or FASTA files just depends on your sequencing platform. I've never heard of anybody really using the quality scores.
  
  user44: FASTA (officially) just stores the name of a sequence and the sequence, unofficially people also add comment fields after the name of the sequence. FASTQ was invented to store both sequence and associated quality values (e.g. from sequencing instruments). SAM was invented to store alignments of (small) sequences (e.g. generated from sequencing) with associated quality values and some further data onto a larger sequences, called reference sequences, the latter being anything from a tiny virus sequence to ultra-large plant sequences.
  
  user27: In a nutshell, 
  FASTA file format is a DNA sequence format for specifying or representing DNA sequences and was first described by Pearson (Pearson,W.R. and Lipman,D.J. (1988) Improved tools for biological sequence comparison. Proc. Natl Acad. Sci. USA, 85, 2444–2448) 
  FASTQ is another DNA sequence file format that extends the FASTA format with the ability to store the sequence quality. The quality scores are often represented in  ASCII characters which correspond to a phred score)
  Both FASTA and FASTQ are common sequence representation formats and have emerged as key data interchange formats for molecular biology and bioinformatics. 
  SAM is format for representing sequence alignment information from a read aligner. It represents sequence information in respect to a given reference sequence. The information is stored in a series of tab delimited ascii columns. The full SAM format specification is available at http://samtools.sourceforge.net/SAM1.pdf 
  
  user29: Let’s start with what they have in common: All three formats store
  
  sequence data, and
  sequence metadata.
  
  Furthermore, all three formats are text-based.
  However, beyond that all three formats are different and serve different purposes.
  Let’s start with the simplest format:
  FASTA
  FASTA stores a variable number of sequence records, and for each record it stores the sequence itself, and a sequence ID. Each record starts with a header line whose first character is >, followed by the sequence ID. The next lines of a record contain the actual sequence.
  The Wikipedia artice gives several examples for peptide sequences, but since FASTQ and SAM are used exclusively (?) for nucleotide sequences, here’s a nucleotide example:
  >Mus_musculus_tRNA-Ala-AGC-1-1 (chr13.trna34-AlaAGC)
  GGGGGTGTAGCTCAGTGGTAGAGCGCGTGCTTAGCATGCACGAGGcCCTGGGTTCGATCC
  CCAGCACCTCCA
  >Mus_musculus_tRNA-Ala-AGC-10-1 (chr13.trna457-AlaAGC)
  GGGGGATTAGCTCAAATGGTAGAGCGCTCGCTTAGCATGCAAGAGGtAGTGGGATCGATG
  CCCACATCCTCCA
  
  The ID can be in any arbitrary format, although several conventions exist.
  In the context of nucleotide sequences, FASTA is mostly used to store reference data; that is, data extracted from a curated database; the above is adapted from GtRNAdb (a database of tRNA sequences).
  FASTQ
  FASTQ was conceived to solve a specific problem arising during sequencing: Due to how different sequencing technologies work, the confidence in each base call (that is, the estimated probability of having correctly identified a given nucleotide) varies. This is expressed in the Phred quality score. FASTA had no standardised way of encoding this. By contrast, a FASTQ record contains a sequence of quality scores for each nucleotide.
  A FASTQ record has the following format:
  
  A line starting with @, containing the sequence ID.
  One or more lines that contain the sequence.
  A new line starting with the character +, and being either empty or repeating the sequence ID.
  One or more lines that contain the quality scores.
  
  Here’s an example of a FASTQ file with two records:
  @071112_SLXA-EAS1_s_7:5:1:817:345
  GGGTGATGGCCGCTGCCGATGGCGTC
  AAATCCCACC
  +
  IIIIIIIIIIIIIIIIIIIIIIIIII
  IIII9IG9IC
  @071112_SLXA-EAS1_s_7:5:1:801:338
  GTTCAGGGATACGACGTTTGTATTTTAAGAATCTGA
  +
  IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII6IBI
  
  FASTQ files are mostly used to store short-read data from high-throughput sequencing experiments. The sequence and quality scores are usually put into a single line each, and indeed many tools assume that each record in a FASTQ file is exactly four lines long, even though this isn’t guaranteed.
  As with FASTA, the format of the sequence ID isn’t standardised, but different producers of FASTQ use fixed notations that follow strict conventions.
  SAM
  SAM files are so complex that a complete description [PDF] takes 15 pages. So here’s the short version.
  The original purpose of SAM files is to store mapping information for sequences from high-throughput sequencing. As a consequence, a SAM record needs to store more than just the sequence and its quality, it also needs to store information about where and how a sequence maps into the reference.
  Unlike the previous formats, SAM is tab-based, and each record, consisting of either 11 or 12 fields, fills exactly one line. Here’s an example (tabs replaced by fixed-width spacing):
  r001  99  chr1  7 30  17M         =  37  39  TTAGATAAAGGATACTG   IIIIIIIIIIIIIIIII
  r002  0   chrX  9 30  3S6M1P1I4M  *  0   0   AAAAGATAAGGATA      IIIIIIIIII6IBI    NM:i:1
  
  For a description of the individual fields, refer to the documentation. The relevant bit is this: SAM can express exactly the same information as FASTQ, plus, as mentioned, the mapping information. However, SAM is also used to store read data without mapping information.
  In addition to sequence records, SAM files can also contain a header, which stores information about the reference that the sequences were mapped to, and the tool used to create the SAM file. Header information precede the sequence records, and consist of lines starting with @.
  SAM itself is almost never used as a storage format; instead, files are stored in BAM format, which is a compact, gzipped, binary representation of SAM. It stores the same information, just more efficiently. And, in conjunction with a search index, allows fast retrieval of individual records from the middle of the file (= fast random access). BAM files are also much more compact than compressed FASTQ or FASTA files.
  
  The above implies a hierarchy in what the formats can store: FASTA ⊂ FASTQ ⊂ SAM.
  In a typical high-throughput analysis workflow, you will encounter all three file types:
  
  FASTA to store the reference genome/transcriptome that the sequence fragments will be mapped to.
  FASTQ to store the sequence fragments before mapping.
  SAM/BAM to store the sequence fragments after mapping.
  
  
original_text: 43: <p>I'd like to learn the differences between 3 common formats such as <a href="https://en.wikipedia.org/wiki/FASTA_format" rel="noreferrer">FASTA</a>, <a href="https://en.wikipedia.org/wiki/FASTQ_format" rel="noreferrer">FASTQ</a> and <a href="https://en.wikipedia.org/wiki/SAM_(file_format)" rel="noreferrer">SAM</a>. How they are different? Are there any benefits of using one over another?</p>
  
  <p>Based on Wikipedia pages, I can't tell the differences between them.</p>
   94: <p>FASTA and FASTQ formats are both file formats that contain sequencing reads while SAM files are these reads aligned to a reference sequence. In other words, FASTA and FASTQ are the &quot;raw data&quot; of sequencing while SAM is the product of aligning the sequencing reads to a refseq.</p>
  <p>A FASTA file contains a read name followed by the sequence. An example of one of these reads for RNASeq might be:</p>
  <pre><code>&gt;Flow cell number: lane number: chip coordinates etc.
  ATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTA
  </code></pre>
  <p>The FASTQ version of this read will have two more lines, one + as a place holder and then a line of quality scores for the base calls. The qualities are given as characters with '!' being the lowest and '~' being the highest, in increasing ASCII value. It would look something like this</p>
  <pre><code>@Flow cell number: lane number: chip coordinates etc.
  ATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTAATTGGCTA
  +
  !''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF&gt;&gt;&gt;&gt;&gt;&gt;CCCCCCC65
  </code></pre>
  <p>A SAM file has many fields for each alignment, the header begins with the @ character. The alignment contains 11 mandatory fields and various optional ones. You can find the spec file here: <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="nofollow noreferrer">https://samtools.github.io/hts-specs/SAMv1.pdf</a> .</p>
  <p>Often you'll see BAM files which are just compressed binary versions of SAM files. You can view these alignment files using various tools, such as SAMtools, IGV or USCS Genome browser.</p>
  <p>As to the benefits, FASTA/FASTQ vs. SAM/BAM is comparing apples and oranges. I do a lot of RNASeq work so generally we take the FASTQ files and align them the a refseq using an aligner such as STAR which outputs SAM/BAM files. There's a lot you can do with just these alignment files, looking at expression, but usually I'll use a tool such as RSEM to &quot;count&quot; the reads from various genes to create an expression matrix, samples as columns and genes as rows. Whether you get FASTQ or FASTA files just depends on your sequencing platform. I've never heard of anybody really using the quality scores.</p>
   44: <p>FASTA (officially) just stores the name of a sequence and the sequence, unofficially people also add comment fields after the name of the sequence. FASTQ was invented to store both sequence and associated quality values (e.g. from sequencing instruments). SAM was invented to store alignments of (small) sequences (e.g. generated from sequencing) with associated quality values and some further data onto a larger sequences, called reference sequences, the latter being anything from a tiny virus sequence to ultra-large plant sequences.</p>
   27: <p>In a nutshell, </p>
  
  <p><code>FASTA</code> file format is a DNA sequence format for specifying or representing DNA sequences and was first described by Pearson <code>(Pearson,W.R. and Lipman,D.J. (1988) Improved tools for biological sequence comparison. Proc. Natl Acad. Sci. USA, 85, 2444–2448)</code> </p>
  
  <p><code>FASTQ</code> is another DNA sequence file format that extends the FASTA format with the ability to store the sequence quality. The quality scores are often represented in  ASCII characters which correspond to a phred score)</p>
  
  <p>Both FASTA and FASTQ are common sequence representation formats and have emerged as key data interchange formats for molecular biology and bioinformatics. </p>
  
  <p><code>SAM</code> is format for representing sequence alignment information from a read aligner. It represents sequence information in respect to a given reference sequence. The information is stored in a series of tab delimited ascii columns. The full SAM format specification is available at <a href="http://samtools.sourceforge.net/SAM1.pdf" rel="noreferrer">http://samtools.sourceforge.net/SAM1.pdf</a> </p>
   29: <p>Let’s start with what they have in common: All three formats store</p>
  <ol>
  <li>sequence data, and</li>
  <li>sequence metadata.</li>
  </ol>
  <p>Furthermore, all three formats are text-based.</p>
  <p>However, beyond that all three formats are different and serve different purposes.</p>
  <p>Let’s start with the simplest format:</p>
  <h3>FASTA</h3>
  <p>FASTA stores a variable number of sequence records, and for each record it stores the sequence itself, and a sequence ID. Each record starts with a header line whose first character is <code>&gt;</code>, followed by the sequence ID. The next lines of a record contain the actual sequence.</p>
  <p>The <a href="https://en.wikipedia.org/wiki/FASTA_format" rel="noreferrer">Wikipedia artice</a> gives several examples for peptide sequences, but since FASTQ and SAM are used exclusively (?) for nucleotide sequences, here’s a nucleotide example:</p>
  <pre><code>&gt;Mus_musculus_tRNA-Ala-AGC-1-1 (chr13.trna34-AlaAGC)
  GGGGGTGTAGCTCAGTGGTAGAGCGCGTGCTTAGCATGCACGAGGcCCTGGGTTCGATCC
  CCAGCACCTCCA
  &gt;Mus_musculus_tRNA-Ala-AGC-10-1 (chr13.trna457-AlaAGC)
  GGGGGATTAGCTCAAATGGTAGAGCGCTCGCTTAGCATGCAAGAGGtAGTGGGATCGATG
  CCCACATCCTCCA
  </code></pre>
  <p>The ID can be in any arbitrary format, although <a href="https://en.wikipedia.org/wiki/FASTA_format#Sequence_identifiers" rel="noreferrer">several conventions exist</a>.</p>
  <p>In the context of nucleotide sequences, FASTA is mostly used to store reference data; that is, data extracted from a curated database; the above is adapted from <a href="http://gtrnadb.ucsc.edu/" rel="noreferrer">GtRNAdb</a> (a database of tRNA sequences).</p>
  <h3>FASTQ</h3>
  <p>FASTQ was conceived to solve a specific problem arising during sequencing: Due to how different sequencing technologies work, the confidence in each <a href="https://biology.stackexchange.com/a/1873/166">base call</a> (that is, the estimated probability of having correctly identified a given nucleotide) varies. This is expressed in the <a href="https://en.wikipedia.org/wiki/Phred_quality_score" rel="noreferrer">Phred quality score</a>. FASTA had no standardised way of encoding this. By contrast, a FASTQ record <a href="https://en.wikipedia.org/wiki/FASTQ_format#Encoding" rel="noreferrer">contains a sequence of quality scores</a> for each nucleotide.</p>
  <p>A FASTQ record has the following format:</p>
  <ol>
  <li>A line starting with <code>@</code>, containing the sequence ID.</li>
  <li>One or more lines that contain the sequence.</li>
  <li>A new line starting with the character <code>+</code>, and being either empty or repeating the sequence ID.</li>
  <li>One or more lines that contain the quality scores.</li>
  </ol>
  <p>Here’s an example of a FASTQ file with two records:</p>
  <pre><code>@071112_SLXA-EAS1_s_7:5:1:817:345
  GGGTGATGGCCGCTGCCGATGGCGTC
  AAATCCCACC
  +
  IIIIIIIIIIIIIIIIIIIIIIIIII
  IIII9IG9IC
  @071112_SLXA-EAS1_s_7:5:1:801:338
  GTTCAGGGATACGACGTTTGTATTTTAAGAATCTGA
  +
  IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII6IBI
  </code></pre>
  <p>FASTQ files are mostly used to store short-read data from high-throughput sequencing experiments. The sequence and quality scores are usually put into a single line each, and indeed many tools assume that each record in a FASTQ file is exactly four lines long, even though this isn’t guaranteed.</p>
  <p>As with FASTA, the format of the sequence ID isn’t standardised, but different producers of FASTQ use <a href="https://en.wikipedia.org/wiki/FASTQ_format#Illumina_sequence_identifiers" rel="noreferrer">fixed notations that follow strict conventions</a>.</p>
  <h3>SAM</h3>
  <p>SAM files are so complex that a <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="noreferrer">complete description</a> <sup>[PDF]</sup> takes 15 pages. So here’s the short version.</p>
  <p>The original purpose of SAM files is to store mapping information for sequences from high-throughput sequencing. As a consequence, a SAM record needs to store more than just the sequence and its quality, it also needs to store information about where and how a sequence maps into the reference.</p>
  <p>Unlike the previous formats, SAM is tab-based, and each record, consisting of either 11 or 12 fields, fills exactly one line. Here’s an example (tabs replaced by fixed-width spacing):</p>
  <pre><code>r001  99  chr1  7 30  17M         =  37  39  TTAGATAAAGGATACTG   IIIIIIIIIIIIIIIII
  r002  0   chrX  9 30  3S6M1P1I4M  *  0   0   AAAAGATAAGGATA      IIIIIIIIII6IBI    NM:i:1
  </code></pre>
  <p>For a description of the individual fields, refer to the documentation. The relevant bit is this: SAM can express exactly the same information as FASTQ, plus, as mentioned, the mapping information. However, SAM is also used to store read data <em>without</em> mapping information.</p>
  <p>In addition to sequence records, SAM files can also contain a <em>header</em>, which stores information about the reference that the sequences were mapped to, and the tool used to create the SAM file. Header information precede the sequence records, and consist of lines starting with <code>@</code>.</p>
  <p>SAM itself is almost never used as a storage format; instead, files are stored in BAM format, which is a compact, gzipped, binary representation of SAM. It stores the same information, just more efficiently. And, in conjunction with a <a href="https://en.wikipedia.org/wiki/Database_index" rel="noreferrer">search index</a>, allows fast retrieval of individual records from the middle of the file (= fast <a href="https://en.wikipedia.org/wiki/Random_access" rel="noreferrer">random access</a>). BAM files are also much more compact than compressed FASTQ or FASTA files.</p>
  <hr />
  <p>The above implies a <em>hierarchy</em> in what the formats can store: FASTA ⊂ FASTQ ⊂ SAM.</p>
  <p>In a typical high-throughput analysis workflow, you will encounter all three file types:</p>
  <ol>
  <li>FASTA to store the reference genome/transcriptome that the sequence fragments will be mapped to.</li>
  <li>FASTQ to store the sequence fragments before mapping.</li>
  <li>SAM/BAM to store the sequence fragments after mapping.</li>
  </ol>
  