text: user2085: Question: Why are there more barcodes than GEMs in 10X chromium data?
  Introduction
  I have several 10X genomics chromium libraries made with the de novo assembly/genome protocol. The 10X chromium device is advertised to produce approximately 1,000,000 GEMs (gel-emulsion droplets). In theory each GEM should have a single barcode that is used to track the provenance of each read pair. If this is true, a single library should have approximately one million barcodes, whether or not it was overloaded or underloaded.
  Results
  The libraries actually have more than one million barcodes, confirmed by longranger basic. Library 1 has 2,369,949 barcodes. Library 2 has 2,311,405 barcodes. Both libraries have approximately 150,000,000 read pairs.
  Why this is a problem
  I'm using the 10X data for some non-standard analyses and want a high number of LinkedReads per molecule (LPM). Having multiple barcodes in a GEM will reduce the number of LPMs per barcode proportional to how many barcodes were present in the GEM.
  Possible explanations
  
  If there are more than approximately one million barcodes in a library then this leaves several possibilities:
  
  
  Each GEM truly has a single barcode. Therefore the Chromium device actually produced significantly more GEMs than advertised.
  Each GEM has a variable number of barcodes. Therefore the number of barcodes is independent of how many GEMs are created.
  Some combination of the above two options. Some variable number of GEMs and a variable number of GEMs per barcode.
  Additional barcodes are created by sequencing errors and misclassification by longranger basic. This would probably manifest in many barcodes that only have one or two linked reads total.
  
  
  Deeper analysis
  I don't know how to test hypotheses 1, 2, & 3 bioinformatically. Hypothesis 4 is testable though by making a histogram of read counts per barcode. To do so, we will use the output of longranger basic for Lib2. 
  Longranger basic outputs an interleaved fastq.gz file that contains the whitelisted barcode in the header. For example (pretending the reads are shorter than 150bp):
  @J00113:232:HFWCKBBXX:4:2224:17756:14889 BX:Z:AAACACCAGACAATAC-1
  AATTACCCAAAATTAATATTTGTATTAATTTTCATATTAATAAC
  +
  JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
  @J00113:232:HFWCKBBXX:4:2224:17756:14889 BX:Z:AAACACCAGACAATAC-1
  CACAATATCAAAAACAAATCAAAAATTAAGAAAATATAACTAAG
  +
  AAF<7FF-JJFJJFJJJJJJAJFJJJFJJJJJFJJJJJJJJJJJ
  
  The BX:Z:AAACACCAGACAATAC-1 tells us that the index is whitelisted (BX:Z), the index sequence (AAACACCAGACAATAC), and the library index for the analysis (-1).
  So, we can count the number of occurrences of each barcode with this code:
  bioawk -cfastx '{print($comment)}' barcoded.fastq.gz | sort | uniq -c > bc_counts.txt
  
  The first column of bc_counts.txt tells us how many individual reads had that barcode. We divide this number by two to see how many read pairs had that barcode and make a histogram.
  awk '{print($1)/2}' barcode_counts.txt | sort | uniq -c | sort -k2 -n | awk '{printf("%d\t%d\n", $2, $1)}' > pairs_per_bc.txt
  
  The first column of this file has the number of read pairs in the barcode, and the second column has how many barcodes have that number of read pairs. We see that there are many barcodes that only have one or two read pairs:
  1       593583
  2       207192
  3       79409
  4       34174
  5       17353
  6       10662
  7       7997
  
  
  It looks like every barcode with fewer than 10 read pairs may be noise. How many barcodes contain at least 10 reads?
  >awk '{if ($1 >= 10){sum += $2} } END{print(sum)}' pairs_per_bc.txt
  1347686
  
  Around one million. 
  Conclusion
  This analysis doesn't give conclusive evidence for the four hypotheses above, but it seems like there is a noise-generating process in 10X data that causes there to be many barcodes with only a few readpairs. 
  
  user235: Your initial guess is almost certainly correct. I don't know about the linked read libraries, but in the 10X single cell sequencing protocol, separating real barcodes from noise barcodes is an important and sensitive step in the analysis pipeline. 
  The current analysis pipeline is to look at a plot, like the one you produced and find the "knees". The plot you produced has two knees, on at around 10 and one at around 200. I'd guess that the ones below 10 are what we call "error barcodes" generated by sequencing is PCR errors from real barcodes, and the ones between 10 and 200 are what we'd call "background" barcodes - in scRNAseq they represent things like GEMs that didn't encapsulate a cell, but did encapsulate some non-cellular DNA or RNA. 
  As there are more than 1,000,000 possible barcodes, not all of them are used.  I believe 10X knows which sequences are used and which are not (ie the barcodes are not random), so I'm surprised that these sequences make it out of the other end of the analysis. 
  
original_text: 2085: <h1>Question: Why are there more barcodes than GEMs in 10X chromium data?</h1>
  
  <h2>Introduction</h2>
  
  <p>I have several 10X genomics chromium libraries <a href="https://assets.ctfassets.net/an68im79xiti/1Jw6vQfW1GOGuO0AsS2gM8/61866afe8c8af5e0eecf6a3d890f58aa/CG00043_GenomeReagentKitv2UserGuide_RevB.pdf" rel="nofollow noreferrer">made with the de novo assembly/genome protocol</a>. The 10X chromium device is <a href="https://community.10xgenomics.com/t5/Genome-Exome-Forum/Question-about-linked-reads-in-smaller-genomes/m-p/58/highlight/true#M9" rel="nofollow noreferrer">advertised to produce approximately 1,000,000 GEMs</a> (gel-emulsion droplets). In theory each GEM should have a single barcode that is used to track the provenance of each read pair. If this is true, a single library should have approximately one million barcodes, whether or not it was overloaded or underloaded.</p>
  
  <h2>Results</h2>
  
  <p>The libraries actually have more than one million barcodes, confirmed by <code>longranger basic</code>. Library 1 has 2,369,949 barcodes. Library 2 has 2,311,405 barcodes. Both libraries have approximately 150,000,000 read pairs.</p>
  
  <h2>Why this is a problem</h2>
  
  <p>I'm using the 10X data for some non-standard analyses and want a high number of <a href="https://community.10xgenomics.com/t5/Genome-Exome-Forum/Question-about-linked-reads-in-smaller-genomes/m-p/58/highlight/true#M9" rel="nofollow noreferrer">LinkedReads per molecule (LPM)</a>. Having multiple barcodes in a GEM will reduce the number of LPMs per barcode proportional to how many barcodes were present in the GEM.</p>
  
  <h2>Possible explanations</h2>
  
  <ul>
  <li>If there are more than approximately one million barcodes in a library then this leaves several possibilities:
  
  <ol>
  <li>Each GEM truly has a single barcode. Therefore the Chromium device actually produced significantly more GEMs than advertised.</li>
  <li>Each GEM has a variable number of barcodes. Therefore the number of barcodes is independent of how many GEMs are created.</li>
  <li>Some combination of the above two options. Some variable number of GEMs and a variable number of GEMs per barcode.</li>
  <li>Additional barcodes are created by sequencing errors and misclassification by <code>longranger basic</code>. This would probably manifest in many barcodes that only have one or two linked reads total.</li>
  </ol></li>
  </ul>
  
  <h2>Deeper analysis</h2>
  
  <p>I don't know how to test hypotheses 1, 2, &amp; 3 bioinformatically. Hypothesis 4 is testable though by making a histogram of read counts per barcode. To do so, we will use the output of <code>longranger basic</code> for Lib2. </p>
  
  <p>Longranger basic outputs an interleaved <code>fastq.gz</code> file that contains the whitelisted barcode in the header. For example (pretending the reads are shorter than 150bp):</p>
  
  <pre><code>@J00113:232:HFWCKBBXX:4:2224:17756:14889 BX:Z:AAACACCAGACAATAC-1
  AATTACCCAAAATTAATATTTGTATTAATTTTCATATTAATAAC
  +
  JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
  @J00113:232:HFWCKBBXX:4:2224:17756:14889 BX:Z:AAACACCAGACAATAC-1
  CACAATATCAAAAACAAATCAAAAATTAAGAAAATATAACTAAG
  +
  AAF&lt;7FF-JJFJJFJJJJJJAJFJJJFJJJJJFJJJJJJJJJJJ
  </code></pre>
  
  <p>The <code>BX:Z:AAACACCAGACAATAC-1</code> <a href="https://support.10xgenomics.com/genome-exome/software/pipelines/latest/output/bam" rel="nofollow noreferrer">tells us that the index is whitelisted</a> (<code>BX:Z</code>), <a href="https://support.10xgenomics.com/genome-exome/software/pipelines/latest/output/bam" rel="nofollow noreferrer">the index sequence</a> (<code>AAACACCAGACAATAC</code>), <a href="https://support.10xgenomics.com/genome-exome/software/pipelines/latest/output/bam" rel="nofollow noreferrer">and the library index for the analysis</a> (<code>-1</code>).</p>
  
  <p>So, we can count the number of occurrences of each barcode with this code:</p>
  
  <pre><code>bioawk -cfastx '{print($comment)}' barcoded.fastq.gz | sort | uniq -c &gt; bc_counts.txt
  </code></pre>
  
  <p>The first column of <code>bc_counts.txt</code> tells us how many individual reads had that barcode. We divide this number by two to see how many read pairs had that barcode and make a histogram.</p>
  
  <pre><code>awk '{print(<span class="math-container">$1)/2}' barcode_counts.txt | sort | uniq -c | sort -k2 -n | awk '{printf("%d\t%d\n", $2, $</span>1)}' &gt; pairs_per_bc.txt
  </code></pre>
  
  <p>The first column of this file has the number of read pairs in the barcode, and the second column has how many barcodes have that number of read pairs. We see that there are many barcodes that only have one or two read pairs:</p>
  
  <pre><code>1       593583
  2       207192
  3       79409
  4       34174
  5       17353
  6       10662
  7       7997
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/pVVuG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pVVuG.png" alt="lib2"></a></p>
  
  <p>It looks like every barcode with fewer than 10 read pairs may be noise. How many barcodes contain at least 10 reads?</p>
  
  <pre><code>&gt;awk '{if (<span class="math-container">$1 &gt;= 10){sum += $</span>2} } END{print(sum)}' pairs_per_bc.txt
  1347686
  </code></pre>
  
  <p>Around one million. </p>
  
  <h2>Conclusion</h2>
  
  <p>This analysis doesn't give conclusive evidence for the four hypotheses above, but it seems like there is a noise-generating process in 10X data that causes there to be many barcodes with only a few readpairs. </p>
   235: <p>Your initial guess is almost certainly correct. I don't know about the linked read libraries, but in the 10X single cell sequencing protocol, separating real barcodes from noise barcodes is an important and sensitive step in the analysis pipeline. </p>
  
  <p>The current analysis pipeline is to look at a plot, like the one you produced and find the "knees". The plot you produced has two knees, on at around 10 and one at around 200. I'd guess that the ones below 10 are what we call "error barcodes" generated by sequencing is PCR errors from real barcodes, and the ones between 10 and 200 are what we'd call "background" barcodes - in scRNAseq they represent things like GEMs that didn't encapsulate a cell, but did encapsulate some non-cellular DNA or RNA. </p>
  
  <p>As there are more than 1,000,000 possible barcodes, not all of them are used.  I believe 10X knows which sequences are used and which are not (ie the barcodes are not random), so I'm surprised that these sequences make it out of the other end of the analysis. </p>
  