text: user7236: I have a files which has columns representing genes,sample numbers and sample names (55). The genes are displayed by occurrence in the samples (highest to lowest-55 and 2).I wish to calculate the genes which may co-occur in samples.
  I tried doing this with awk 'n=x[$1,$2,$3,$4,$5,$6]{print n"\n"$0;} {x[$1,$2,$3,$4,$5,$6]=$0;}' adding from 1 to 55 columns, but it doesn't work.
  An example here:
  Nfib    55  A1.S1   A2.S2   A3.S3   A4.S4   A6.S6   A7.S7   A8.S8   A9.S9   A9.S9*  A10.S10 A11.S11 A12.S12 IZ_BH_2.1   IZ_BH_2.2   IZ_BH_2.3   IZ_BH_2.4   IZ_BH_2.5   IZ_BH_2.6   IZ_BH_2.7   IZ_BH_2.8   IZ_BH_2.9   IZ_BH_2.10  IZ_BH_2.11  IZ_BH_2.12  IZ_BH_3.1   IZ_BH_3.2   IZ_BH_3.3   IZ_BH_3.4   IZ_BH_3.5   IZ_BH_3.6   IZ_BH_3.7   IZ_BH_3.8   IZ_BH_3.9   IZ_BH_3.10  IZ_BH_3.11  IZ_BH_3.11* IZ_BH_3.12  IZ_BH_4.1   IZ_BH_4.2   IZ_BH_4.4   IZ_BH_4.5   IZ_BH_4.6   IZ_BH_4.7   IZ_BH_4.8   IZ_BH_4.9   IZ_BH_4.10  IZ_BH_4.11  IZ_BH_5.1   IZ_BH_5.2   IZ_BH_5.3   IZ_BH_5.4   IZ_BH_5.5   IZ_BH_5.6   IZ_BH_5.7   IZ_BH_5.8
  
  
  Lyn 41  A1.S1   A4.S4   A7.S7   A8.S8   A9.S9   A9.S9*  A10.S10 A11.S11 IZ_BH_2.1   IZ_BH_2.2   IZ_BH_2.3   IZ_BH_2.4   IZ_BH_2.5   IZ_BH_2.6   IZ_BH_2.7   IZ_BH_2.9   IZ_BH_2.11  IZ_BH_2.12  IZ_BH_3.2   IZ_BH_3.3   IZ_BH_3.4   IZ_BH_3.5   IZ_BH_3.6   IZ_BH_3.8   IZ_BH_3.9   IZ_BH_3.10  IZ_BH_3.11  IZ_BH_4.1   IZ_BH_4.2   IZ_BH_4.3   IZ_BH_4.4   IZ_BH_4.7   IZ_BH_4.8   IZ_BH_4.11  IZ_BH_5.1   IZ_BH_5.3   IZ_BH_5.4   IZ_BH_5.5   IZ_BH_5.6   IZ_BH_5.7   IZ_BH_5.8
  
  
  Cdk5rap2    52  A1.S1   A2.S2   A3.S3   A6.S6   A7.S7   A8.S8   A9.S9   A9.S9*  A11.S11 A12.S12 IZ_BH_2.1   IZ_BH_2.2   IZ_BH_2.3   IZ_BH_2.4   IZ_BH_2.5   IZ_BH_2.6   IZ_BH_2.7   IZ_BH_2.8   IZ_BH_2.9   IZ_BH_2.10  IZ_BH_2.11  IZ_BH_2.12  IZ_BH_3.1   IZ_BH_3.2   IZ_BH_3.3   IZ_BH_3.4   IZ_BH_3.5   IZ_BH_3.6   IZ_BH_3.7   IZ_BH_3.8   IZ_BH_3.9   IZ_BH_3.10  IZ_BH_3.11  IZ_BH_3.11* IZ_BH_3.12  IZ_BH_4.1   IZ_BH_4.2   IZ_BH_4.3   IZ_BH_4.4   IZ_BH_4.6   IZ_BH_4.8   IZ_BH_4.9   IZ_BH_4.10  IZ_BH_4.11  IZ_BH_5.1   IZ_BH_5.2   IZ_BH_5.3   IZ_BH_5.4   IZ_BH_5.5   IZ_BH_5.6   IZ_BH_5.7   IZ_BH_5.8
  
  The first name is of the gene, followed by number of samples and sample names.
  for these 3 genes these are the commonly occuring samples:
  A1.S1, A7.S7, A8.S8, A9.S9*, A9.S9, A11.S11, IZ_BH_2.1, IZ_BH_2.2, IZ_BH_2.3,IZ_BH_2.4, IZ_BH_2.5,IZ_BH_2.6,IZ_BH_2.7, IZ_BH_2.9, IZ_BH_2.11,IZ_BH_2.12, IZ_BH_3.2,IZ_BH_3.3,IZ_BH_3.4, IZ_BH_3.5, IZ_BH_3.6,IZ_BH_3.8,
  IZ_BH_3.9,IZ_BH_3.10,IZ_BH_3.11,IZ_BH_4.1,IZ_BH_4.2,IZ_BH_4.4,IZ_BH_4.8,IZ_BH_4.11,IZ_BH_5.1,IZ_BH_5.3,IZ_BH_5.4,IZ_BH_5.5,IZ_BH_5.6,IZ_BH_5.7,IZ_BH_5.8
  
  Kindly help.
  
  user439: I think what makes this especially hard is the way your data is formatted.
  The problem would be way easier if you had a matrix of presence / absence instead. Each row would be a gene, each column a sample, and the values would be 0 (absent) or 1 (present).
  Awk becomes a bit inconvenient for this task, so I'll use python instead. To get such a matrix from your file, you could do the following:
  import numpy as np
  import pandas as pd
  
  # Get total number of genes / samples
  all_samples = set()
  all_genes = set()
  with open('genes.txt', 'r') as genes:
    for line in genes:
      line = line.split('\t')
      all_genes.add(line[0])
      all_samples.update(line[2:])
  all_samples = list(all_samples)
  all_genes = list(all_genes)
  
  # Initialize presence matrix with correct dimensions
  pres_mat = np.zeros((len(all_genes), len(all_samples)), dtype=int)
  
  # Map each sample/gene to a row/col
  row_map = {gene: i for i, gene in enumerate(all_genes)}
  col_map = {sample: i for i, sample in enumerate(all_samples)}
  
  # Fill matrix incrementally
  with open('genes.txt', 'r') as genes:
    for line in genes:
      line = line.split('\t')
      curr_gene = line[0]
      row = row_map[curr_gene]
      for sample in line[2:]:
        col = col_map[sample]
        pres_mat[row, col] = 1
  
  # Add names and write to file
  df = pd.DataFrame(pres_mat, columns=all_samples)
  df.insert(0, 'gene', all_genes)
  df.to_csv('gene_presence.tsv, sep='\t')
  
  
  You should get something that looks like:
         gene  IZ_BH_5.5  A8.S8  IZ_BH_2.1  A9.S9*  IZ_BH_5.3  A9.S9  ...  IZ_BH_3.11  IZ_BH_4.7  IZ_BH_4.9  IZ_BH_2.8  IZ_BH_3.5  IZ_BH_2.6  IZ_BH_4.3
  0       Lyn          1      1          1       1          1      1  ...           1          1          0          0          1          1          1
  1      Nfib          1      1          1       1          1      1  ...           1          1          1          1          1          1          0
  2  Cdk5rap2          1      1          1       1          1      1  ...           1          0          1          1          1          1          1
  
  With this format, it is now easy to compute the most frequent samples. E.g. df.sum(axis=0) to get the number of occurence of each sample for these 3 genes. It would also allow you to do different things, such as computing the pairwise similarity between samples.
  
original_text: 7236: <p>I have a files which has columns representing genes,sample numbers and sample names (55). The genes are displayed by occurrence in the samples (highest to lowest-55 and 2).I wish to calculate the genes which may co-occur in samples.
  I tried doing this with <code>awk 'n=x[$1,$2,$3,$4,$5,$6]{print n&quot;\n&quot;$0;} {x[$1,$2,$3,$4,$5,$6]=$0;}'</code> adding from 1 to 55 columns, but it doesn't work.</p>
  <p>An example here:</p>
  <pre><code>Nfib    55  A1.S1   A2.S2   A3.S3   A4.S4   A6.S6   A7.S7   A8.S8   A9.S9   A9.S9*  A10.S10 A11.S11 A12.S12 IZ_BH_2.1   IZ_BH_2.2   IZ_BH_2.3   IZ_BH_2.4   IZ_BH_2.5   IZ_BH_2.6   IZ_BH_2.7   IZ_BH_2.8   IZ_BH_2.9   IZ_BH_2.10  IZ_BH_2.11  IZ_BH_2.12  IZ_BH_3.1   IZ_BH_3.2   IZ_BH_3.3   IZ_BH_3.4   IZ_BH_3.5   IZ_BH_3.6   IZ_BH_3.7   IZ_BH_3.8   IZ_BH_3.9   IZ_BH_3.10  IZ_BH_3.11  IZ_BH_3.11* IZ_BH_3.12  IZ_BH_4.1   IZ_BH_4.2   IZ_BH_4.4   IZ_BH_4.5   IZ_BH_4.6   IZ_BH_4.7   IZ_BH_4.8   IZ_BH_4.9   IZ_BH_4.10  IZ_BH_4.11  IZ_BH_5.1   IZ_BH_5.2   IZ_BH_5.3   IZ_BH_5.4   IZ_BH_5.5   IZ_BH_5.6   IZ_BH_5.7   IZ_BH_5.8
  
  
  Lyn 41  A1.S1   A4.S4   A7.S7   A8.S8   A9.S9   A9.S9*  A10.S10 A11.S11 IZ_BH_2.1   IZ_BH_2.2   IZ_BH_2.3   IZ_BH_2.4   IZ_BH_2.5   IZ_BH_2.6   IZ_BH_2.7   IZ_BH_2.9   IZ_BH_2.11  IZ_BH_2.12  IZ_BH_3.2   IZ_BH_3.3   IZ_BH_3.4   IZ_BH_3.5   IZ_BH_3.6   IZ_BH_3.8   IZ_BH_3.9   IZ_BH_3.10  IZ_BH_3.11  IZ_BH_4.1   IZ_BH_4.2   IZ_BH_4.3   IZ_BH_4.4   IZ_BH_4.7   IZ_BH_4.8   IZ_BH_4.11  IZ_BH_5.1   IZ_BH_5.3   IZ_BH_5.4   IZ_BH_5.5   IZ_BH_5.6   IZ_BH_5.7   IZ_BH_5.8
  
  
  Cdk5rap2    52  A1.S1   A2.S2   A3.S3   A6.S6   A7.S7   A8.S8   A9.S9   A9.S9*  A11.S11 A12.S12 IZ_BH_2.1   IZ_BH_2.2   IZ_BH_2.3   IZ_BH_2.4   IZ_BH_2.5   IZ_BH_2.6   IZ_BH_2.7   IZ_BH_2.8   IZ_BH_2.9   IZ_BH_2.10  IZ_BH_2.11  IZ_BH_2.12  IZ_BH_3.1   IZ_BH_3.2   IZ_BH_3.3   IZ_BH_3.4   IZ_BH_3.5   IZ_BH_3.6   IZ_BH_3.7   IZ_BH_3.8   IZ_BH_3.9   IZ_BH_3.10  IZ_BH_3.11  IZ_BH_3.11* IZ_BH_3.12  IZ_BH_4.1   IZ_BH_4.2   IZ_BH_4.3   IZ_BH_4.4   IZ_BH_4.6   IZ_BH_4.8   IZ_BH_4.9   IZ_BH_4.10  IZ_BH_4.11  IZ_BH_5.1   IZ_BH_5.2   IZ_BH_5.3   IZ_BH_5.4   IZ_BH_5.5   IZ_BH_5.6   IZ_BH_5.7   IZ_BH_5.8
  </code></pre>
  <p>The first name is of the gene, followed by number of samples and sample names.</p>
  <p>for these 3 genes these are the commonly occuring samples:</p>
  <pre><code>A1.S1, A7.S7, A8.S8, A9.S9*, A9.S9, A11.S11, IZ_BH_2.1, IZ_BH_2.2, IZ_BH_2.3,IZ_BH_2.4, IZ_BH_2.5,IZ_BH_2.6,IZ_BH_2.7, IZ_BH_2.9, IZ_BH_2.11,IZ_BH_2.12, IZ_BH_3.2,IZ_BH_3.3,IZ_BH_3.4, IZ_BH_3.5, IZ_BH_3.6,IZ_BH_3.8,
  IZ_BH_3.9,IZ_BH_3.10,IZ_BH_3.11,IZ_BH_4.1,IZ_BH_4.2,IZ_BH_4.4,IZ_BH_4.8,IZ_BH_4.11,IZ_BH_5.1,IZ_BH_5.3,IZ_BH_5.4,IZ_BH_5.5,IZ_BH_5.6,IZ_BH_5.7,IZ_BH_5.8
  </code></pre>
  <p>Kindly help.</p>
   439: <p>I think what makes this especially hard is the way your data is formatted.
  The problem would be way easier if you had a matrix of presence / absence instead. Each row would be a gene, each column a sample, and the values would be 0 (absent) or 1 (present).</p>
  <p>Awk becomes a bit inconvenient for this task, so I'll use python instead. To get such a matrix from your file, you could do the following:</p>
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  import pandas as pd
  
  # Get total number of genes / samples
  all_samples = set()
  all_genes = set()
  with open('genes.txt', 'r') as genes:
    for line in genes:
      line = line.split('\t')
      all_genes.add(line[0])
      all_samples.update(line[2:])
  all_samples = list(all_samples)
  all_genes = list(all_genes)
  
  # Initialize presence matrix with correct dimensions
  pres_mat = np.zeros((len(all_genes), len(all_samples)), dtype=int)
  
  # Map each sample/gene to a row/col
  row_map = {gene: i for i, gene in enumerate(all_genes)}
  col_map = {sample: i for i, sample in enumerate(all_samples)}
  
  # Fill matrix incrementally
  with open('genes.txt', 'r') as genes:
    for line in genes:
      line = line.split('\t')
      curr_gene = line[0]
      row = row_map[curr_gene]
      for sample in line[2:]:
        col = col_map[sample]
        pres_mat[row, col] = 1
  
  # Add names and write to file
  df = pd.DataFrame(pres_mat, columns=all_samples)
  df.insert(0, 'gene', all_genes)
  df.to_csv('gene_presence.tsv, sep='\t')
  
  </code></pre>
  <p>You should get something that looks like:</p>
  <pre><code>       gene  IZ_BH_5.5  A8.S8  IZ_BH_2.1  A9.S9*  IZ_BH_5.3  A9.S9  ...  IZ_BH_3.11  IZ_BH_4.7  IZ_BH_4.9  IZ_BH_2.8  IZ_BH_3.5  IZ_BH_2.6  IZ_BH_4.3
  0       Lyn          1      1          1       1          1      1  ...           1          1          0          0          1          1          1
  1      Nfib          1      1          1       1          1      1  ...           1          1          1          1          1          1          0
  2  Cdk5rap2          1      1          1       1          1      1  ...           1          0          1          1          1          1          1
  </code></pre>
  <p>With this format, it is now easy to compute the most frequent samples. E.g. <code>df.sum(axis=0)</code> to get the number of occurence of each sample for these 3 genes. It would also allow you to do different things, such as computing the pairwise similarity between samples.</p>
  