text: user2621: I’m trying to develop a table from a series of lists generated using the Immunarch library to process TCR sequencing results. Each list is comprised of CDR3.aa (clone) information which are character strings and their count in a particular sample. The clones are short but vary between 7 and 20 (or more).  Each list has a header that identifies the sample sequenced. I have a list of 66 samples. Each sample can have several thousand clone strings. Not every clone is contained in every sample so the number of clones listed in the samples varies.  Here’s an example of the structure of a single sample list.
  $PreAg_18_2
  CDR3.aa           Count
  CASSYGTAYTGELFF   1623
  CASSRGDSDNSPLHF   1440
  CASSREKAFF        1161
  CSGMGALAKNIQYF     949
  CSAYTGLSYEQYF      813
  CASSLSLAVNSPLHF    634
  CAIRDTPGSPQHF      574
  CATGQVNTEAFF       555
  CASSLKGQGGSPLHF    499
  CASSYSRSPQPQHF     478
  
  I want to combine the results in a single table showing the clone counts with all the clone strings listed on the y-axis and the sample ID listed on the x-axis. For example:  
                  10_pep_10_1     preAg_10_2      Dec_2_18_1  …... 
  CASSYGTAYTGELFF    1623         234             0
  CASSRGDSDNSPLHF    1440         522             28
  CASSREKAFF         1161         445             50  
  CSGMGALAKNIQYF      949         24              0
  CASSYSRSPQPQHF      478         0               398
  .
  .
  
  Currently, I'm working with R because that's what Immunarch is in, but I wonder if a) There's a better way to do this in Python, or b) If it's doable in r, then how do I generate the matrix matches.  The main problem is that not all the samples have the same number of clones, so I first have to separate each sample data before I can convert it to a data frame, and then how can I match the actual count with the actual clone.  I can't just merge the data frames.  Any suggestion is greatly appreciated.
  Here's what i have for code right now.
  Code:
      library(immunarch)
      library(stringr)
      library(plyr)
  immdata = repLoad("/mnt/data/Development/Analysis_Scripts/input_files/")
  
  all <- immdata$data
  
  # Get list headers (names) and convert to df
  sample.id <- names(all)
  sample.id <- data.frame(sample.id)
  
  # Get list of clones and filter for unique clones per list.
  for (i in 1:length(all)){
      all[[i]]$Sample.ID<-names(all)[i]
      all[[i]]<-all[[i]][,c("CDR3.aa", "Clones")]
  }
  
  # all is the variable that contains the list of samples and clones.
  all.split <- split(all, )
  
  # make vector of all clones
  all.clones <- unlist(all, use.names=FALSE)
  
  # Removes clone repeats
  all.clones.u <- unique(all.clones)
  
  # convert list of clones to data frame
  all.clones.u <- data.frame(all.clones.u)
  
  At this point I have both the list of sample ids and the list of clones.  My problem is now matching each count with respective sample and  clone in the table.  Any suggestions?
  
  user16818: That can be easily done with MiXCR software.
  If you first analyze the raw data using mixcr, which can be easily done with mixcr analyze command depending on your data structure:
  e.g.
  mixcr analyze generic-tcr-amplicon \
  --species mmu \
  --rna \
  --rigid-left-alignment-boundary \
  --floating-right-alignment-boundary C \
    input_R1.fastq.gz \
    input_R2.fastq.gz \
    result
  
  More on that here:
  https://docs.milaboratories.com/mixcr/reference/mixcr-analyze/
  After that you will have .clns file for every sample.
  Then, to merge all samples you can use mixcr exportClonesOverlap and all .clns files as an input, e.g.:
  mixcr exportClonesOverlap \
    --criteria "CDR3|AA|V|J" \
    clonesets/*.clns
    overlapTable.tsv 
  
  --criteria allows you to specify the criteria required for clonotypes to be treated as identical. "CDR3|AA|V|J" means that two clonotypes have to share cdr3 amino acid sequence, v gene and j gene.
  overlapTable.tsv is exactly what you need.
  There are other parameters you can tune (like export only productive clonotypes etc.), more on that here:
  https://docs.milaboratories.com/mixcr/reference/mixcr-exportClonesOverlap/#example
  
  user3541: You could give a try with dplyr's full_join() and purrr's reduce(). full_join() will make sure that you won't lose any data and reduce() will merge tables in your list from left to right, left being the first table of the list in the first iteration but would become merged table1+table2 in the second iteration and so on. This left to right "reduction" will also be helpful when renaming the columns of the resulting table.
  Without your workspace I have attempted to replicate your data: There is a list of 3 data frames, each with 2 columns:
  library(dplyr)
  library(purrr)
  
  df1 <- data.frame(CDR3.aa = c("A", "B", "C"),
                    Count = c(10,20,100))
  
  df2 <- data.frame(CDR3.aa = c("A", "B", "D"),
                    Count = c(100,200,300))
  
  df3 <- data.frame(CDR3.aa = c("D", "E", "F"),
                    Count = c(10,20,100))
  
  my_list <- list(df1 = df1, df2 = df2, df3 = df3)
  
  > my_list
  $df1
    CDR3.aa Count
  1       A    10
  2       B    20
  3       C   100
  
  $df2
    CDR3.aa Count
  1       A   100
  2       B   200
  3       D   300
  
  $df3
    CDR3.aa Count
  1       D    10
  2       E    20
  3       F   100
  
  # "by" argument specifies which column to use when merging
  my_table <- reduce(my_list, full_join, by = "CDR3.aa")
  
  names(my_table) <- c("CDR3.aa", names(my_list))
  
  > my_table
    CDR3.aa df1 df2 df3
  1       A  10 100  NA
  2       B  20 200  NA
  3       C 100  NA  NA
  4       D  NA 300  10
  5       E  NA  NA  20
  6       F  NA  NA 100
  
  Clonotypes that are not present in table will be "NA".
  
original_text: 2621: <p>I’m trying to develop a table from a series of lists generated using the Immunarch library to process TCR sequencing results. Each list is comprised of CDR3.aa (clone) information which are character strings and their count in a particular sample. The clones are short but vary between 7 and 20 (or more).  Each list has a header that identifies the sample sequenced. I have a list of 66 samples. Each sample can have several thousand clone strings. Not every clone is contained in every sample so the number of clones listed in the samples varies.  Here’s an example of the structure of a single sample list.</p>
  
  <pre><code>$PreAg_18_2
  CDR3.aa           Count
  CASSYGTAYTGELFF   1623
  CASSRGDSDNSPLHF   1440
  CASSREKAFF        1161
  CSGMGALAKNIQYF     949
  CSAYTGLSYEQYF      813
  CASSLSLAVNSPLHF    634
  CAIRDTPGSPQHF      574
  CATGQVNTEAFF       555
  CASSLKGQGGSPLHF    499
  CASSYSRSPQPQHF     478
  </code></pre>
  
  <p>I want to combine the results in a single table showing the clone counts with all the clone strings listed on the y-axis and the sample ID listed on the x-axis. For example:  </p>
  
  <pre><code>                10_pep_10_1     preAg_10_2      Dec_2_18_1  …... 
  CASSYGTAYTGELFF    1623         234             0
  CASSRGDSDNSPLHF    1440         522             28
  CASSREKAFF         1161         445             50  
  CSGMGALAKNIQYF      949         24              0
  CASSYSRSPQPQHF      478         0               398
  .
  .
  </code></pre>
  
  <p>Currently, I'm working with R because that's what Immunarch is in, but I wonder if a) There's a better way to do this in Python, or b) If it's doable in r, then how do I generate the matrix matches.  The main problem is that not all the samples have the same number of clones, so I first have to separate each sample data before I can convert it to a data frame, and then how can I match the actual count with the actual clone.  I can't just merge the data frames.  Any suggestion is greatly appreciated.</p>
  
  <p>Here's what i have for code right now.
  Code:
      library(immunarch)
      library(stringr)
      library(plyr)</p>
  
  <pre><code>immdata = repLoad("/mnt/data/Development/Analysis_Scripts/input_files/")
  
  all &lt;- immdata$data
  
  # Get list headers (names) and convert to df
  sample.id &lt;- names(all)
  sample.id &lt;- data.frame(sample.id)
  
  # Get list of clones and filter for unique clones per list.
  for (i in 1:length(all)){
      all[[i]]$Sample.ID&lt;-names(all)[i]
      all[[i]]&lt;-all[[i]][,c("CDR3.aa", "Clones")]
  }
  
  # all is the variable that contains the list of samples and clones.
  all.split &lt;- split(all, )
  
  # make vector of all clones
  all.clones &lt;- unlist(all, use.names=FALSE)
  
  # Removes clone repeats
  all.clones.u &lt;- unique(all.clones)
  
  # convert list of clones to data frame
  all.clones.u &lt;- data.frame(all.clones.u)
  </code></pre>
  
  <p>At this point I have both the list of sample ids and the list of clones.  My problem is now matching each count with respective sample and  clone in the table.  Any suggestions?</p>
   16818: <p>That can be easily done with MiXCR software.
  If you first analyze the raw data using mixcr, which can be easily done with <code>mixcr analyze</code> command depending on your data structure:</p>
  <p>e.g.</p>
  <pre><code>mixcr analyze generic-tcr-amplicon \
  --species mmu \
  --rna \
  --rigid-left-alignment-boundary \
  --floating-right-alignment-boundary C \
    input_R1.fastq.gz \
    input_R2.fastq.gz \
    result
  </code></pre>
  <p>More on that here:</p>
  <p><a href="https://docs.milaboratories.com/mixcr/reference/mixcr-analyze/" rel="nofollow noreferrer">https://docs.milaboratories.com/mixcr/reference/mixcr-analyze/</a></p>
  <p>After that you will have .clns file for every sample.</p>
  <p>Then, to merge all samples you can use <code>mixcr exportClonesOverlap</code> and all .clns files as an input, e.g.:</p>
  <pre><code>mixcr exportClonesOverlap \
    --criteria &quot;CDR3|AA|V|J&quot; \
    clonesets/*.clns
    overlapTable.tsv 
  </code></pre>
  <p><code>--criteria</code> allows you to specify the criteria required for clonotypes to be treated as identical. <code>&quot;CDR3|AA|V|J&quot;</code> means that two clonotypes have to share cdr3 amino acid sequence, v gene and j gene.</p>
  <p>overlapTable.tsv is exactly what you need.</p>
  <p>There are other parameters you can tune (like export only productive clonotypes etc.), more on that here:</p>
  <p><a href="https://docs.milaboratories.com/mixcr/reference/mixcr-exportClonesOverlap/#example" rel="nofollow noreferrer">https://docs.milaboratories.com/mixcr/reference/mixcr-exportClonesOverlap/#example</a></p>
   3541: <p>You could give a try with dplyr's <code>full_join()</code> and purrr's <code>reduce()</code>. <code>full_join()</code> will make sure that you won't lose any data and <code>reduce()</code> will merge tables in your list from left to right, left being the first table of the list in the first iteration but would become merged <em>table1+table2</em> in the second iteration and so on. This left to right "reduction" will also be helpful when renaming the columns of the resulting table.</p>
  
  <p>Without your workspace I have attempted to replicate your data: There is a list of 3 data frames, each with 2 columns:</p>
  
  <pre><code>library(dplyr)
  library(purrr)
  
  df1 &lt;- data.frame(CDR3.aa = c("A", "B", "C"),
                    Count = c(10,20,100))
  
  df2 &lt;- data.frame(CDR3.aa = c("A", "B", "D"),
                    Count = c(100,200,300))
  
  df3 &lt;- data.frame(CDR3.aa = c("D", "E", "F"),
                    Count = c(10,20,100))
  
  my_list &lt;- list(df1 = df1, df2 = df2, df3 = df3)
  
  &gt; my_list
  $df1
    CDR3.aa Count
  1       A    10
  2       B    20
  3       C   100
  
  $df2
    CDR3.aa Count
  1       A   100
  2       B   200
  3       D   300
  
  $df3
    CDR3.aa Count
  1       D    10
  2       E    20
  3       F   100
  
  # "by" argument specifies which column to use when merging
  my_table &lt;- reduce(my_list, full_join, by = "CDR3.aa")
  
  names(my_table) &lt;- c("CDR3.aa", names(my_list))
  
  &gt; my_table
    CDR3.aa df1 df2 df3
  1       A  10 100  NA
  2       B  20 200  NA
  3       C 100  NA  NA
  4       D  NA 300  10
  5       E  NA  NA  20
  6       F  NA  NA 100
  </code></pre>
  
  <p>Clonotypes that are not present in table will be "NA".</p>
  