text: user6670: I am trying to make a paired matrix of gene-gene correlation. Considering that I have a huge matrix (13000 genes and 900 samples) and for some reasons I don't want to decrease the number of my genes, my gene-correlation matrix would be 13000*13000 and my paired matrix will become 169 million *4 (Column 1: Gene 1; Column 2 : Gene 2; Column 3: Correlations; Column 4: P-values) . In this case, I have to exclude unnecessary calculations as much as I can. I have excluded the situation that Gene 1 = Gene 2. But I couldn't find a way to exclude the condition that "Column 1: Gene 1 ; Column 2: Gene 2 = Column 1: Gene 2; Column 2: Gene 1 ". To make a long story short, correlation between G1 and G2 is equal to G2 and G1. It is like calculating just lower section of diag in a symmetric matrix. I would be grateful if anybody help me in this case. I have enclosed my python codes here:
  import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import spearmanr
  
  
  din=pd.read_csv('m_test.csv', index_col=0)
  
  out=pd.DataFrame()
  outdf=pd.DataFrame()
  
  for g1 in din.index:
      for g2 in din.index:
          temp=din.loc[[g1, g2]]
          if g1==g2:
              next
          else: 
              spR, spP=spearmanr(temp.loc[g1], temp.loc[g2])
              frame={'g1':[g1], 'g2':[g2], 'spR':[spR], 'spP':[spP]}
  
              out=pd.DataFrame(frame)
  
              outdf=pd.concat([outdf, out])
  
  
  user650: You don't need g2 to go from 0 to din.index, just from 0 to g1 - 1. This way, you'll end up calculating just for the "lower triangle":
  g1-> 0   1   2   3
  g2
  0                    #(0,0 to 0,1-1 = nothing)
  1    X               #(1,0 to 1,1-1 = 1,0)
  2    X   X           #(2,0 to 2,2-1 = 2,0; 2,1)
  3    X   X   X       #(3,0 to 3,3-1 = 3,0; 3,1 and 3,2)
  4    X   X   X   X   #(4,0 to 4,4-1 = 4,0; 4,1; 4,2 and 4,3)
  
  
  user6670: Actually I found an easy and efficient way.
  import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import pearsonr
  from scipy.stats import spearmanr
  from scipy.stats import kendalltau
  
  pro=pd.read_csv('m_test.csv', index_col=0)
  
  xInds = []
  yInds = []
  for i in range(len(pro.index)):
      for j in range(i+1, len(pro.index)):
          xInds.append(i)
          yInds.append(j)
  
  z=0
  Rlist_sp = []
  Plist_sp = []
  Rlist_pe = []
  Plist_pe = []
  while z < len(xInds):
      b=xInds[z]
      c=yInds[z]
  
      spR, spP = spearmanr(pro.iloc[b].values, pro.iloc[c].values)
      peR, peP = pearsonr(pro.iloc[b].values, pro.iloc[c].values) 
  
      Rlist_sp.append(spR)
      Plist_sp.append(spP)
      Rlist_pe.append(peR)
      Plist_pe.append(peP)    
      z=z+1 
  
  G1=pd.DataFrame(xInds)
  G2=pd.DataFrame(yInds)
  R_sp=pd.DataFrame(Rlist_sp)
  P_sp=pd.DataFrame(Plist_sp)
  R_pe=pd.DataFrame(Rlist_pe)
  P_pe=pd.DataFrame(Plist_pe)        
  
  Final=pd.concat([G1, G2, R_sp, P_sp, R_pe, P_pe], axis=1)
  np.savetxt('Final_gg_A549.txt', Final.values, fmt='%s', delimiter='\t')  
  
  
original_text: 6670: <p>I am trying to make a paired matrix of gene-gene correlation. Considering that I have a huge matrix (13000 genes and 900 samples) and for some reasons I don't want to decrease the number of my genes, my gene-correlation matrix would be 13000*13000 and my paired matrix will become 169 million *4 (Column 1: Gene 1; Column 2 : Gene 2; Column 3: Correlations; Column 4: P-values) . In this case, I have to exclude unnecessary calculations as much as I can. I have excluded the situation that Gene 1 = Gene 2. But I couldn't find a way to exclude the condition that "Column 1: Gene 1 ; Column 2: Gene 2 = Column 1: Gene 2; Column 2: Gene 1 ". To make a long story short, correlation between G1 and G2 is equal to G2 and G1. It is like calculating just lower section of diag in a symmetric matrix. I would be grateful if anybody help me in this case. I have enclosed my python codes here:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import spearmanr
  
  
  din=pd.read_csv('m_test.csv', index_col=0)
  
  out=pd.DataFrame()
  outdf=pd.DataFrame()
  
  for g1 in din.index:
      for g2 in din.index:
          temp=din.loc[[g1, g2]]
          if g1==g2:
              next
          else: 
              spR, spP=spearmanr(temp.loc[g1], temp.loc[g2])
              frame={'g1':[g1], 'g2':[g2], 'spR':[spR], 'spP':[spP]}
  
              out=pd.DataFrame(frame)
  
              outdf=pd.concat([outdf, out])
  </code></pre>
   650: <p>You don't need <code>g2</code> to go from <code>0</code> to <code>din.index</code>, just from <code>0</code> to <code>g1 - 1</code>. This way, you'll end up calculating just for the "lower triangle":</p>
  
  <pre><code>g1-&gt; 0   1   2   3
  g2
  0                    #(0,0 to 0,1-1 = nothing)
  1    X               #(1,0 to 1,1-1 = 1,0)
  2    X   X           #(2,0 to 2,2-1 = 2,0; 2,1)
  3    X   X   X       #(3,0 to 3,3-1 = 3,0; 3,1 and 3,2)
  4    X   X   X   X   #(4,0 to 4,4-1 = 4,0; 4,1; 4,2 and 4,3)
  </code></pre>
   6670: <p>Actually I found an easy and efficient way.</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import pearsonr
  from scipy.stats import spearmanr
  from scipy.stats import kendalltau
  
  pro=pd.read_csv('m_test.csv', index_col=0)
  
  xInds = []
  yInds = []
  for i in range(len(pro.index)):
      for j in range(i+1, len(pro.index)):
          xInds.append(i)
          yInds.append(j)
  
  z=0
  Rlist_sp = []
  Plist_sp = []
  Rlist_pe = []
  Plist_pe = []
  while z &lt; len(xInds):
      b=xInds[z]
      c=yInds[z]
  
      spR, spP = spearmanr(pro.iloc[b].values, pro.iloc[c].values)
      peR, peP = pearsonr(pro.iloc[b].values, pro.iloc[c].values) 
  
      Rlist_sp.append(spR)
      Plist_sp.append(spP)
      Rlist_pe.append(peR)
      Plist_pe.append(peP)    
      z=z+1 
  
  G1=pd.DataFrame(xInds)
  G2=pd.DataFrame(yInds)
  R_sp=pd.DataFrame(Rlist_sp)
  P_sp=pd.DataFrame(Plist_sp)
  R_pe=pd.DataFrame(Rlist_pe)
  P_pe=pd.DataFrame(Plist_pe)        
  
  Final=pd.concat([G1, G2, R_sp, P_sp, R_pe, P_pe], axis=1)
  np.savetxt('Final_gg_A549.txt', Final.values, fmt='%s', delimiter='\t')  
  </code></pre>
  