title: Calculating distance in matrices Pandas Python
body: <p>Trying to calculate Pearson Coefficient for a big matrix (e.g. 300000x12000)</p>
  
  <p>So I'm doing next:</p>
  
  <p>Import libraries:</p>
  
  <pre><code>import pandas as pd
  from scipy.spatial.distance import cosine
  from scipy.stats.stats import pearsonr
  </code></pre>
  
  <p>Do the dummy table using pd.get_dummies:</p>
  
  <pre><code>table=pd.get_dummies(data['word'])
  </code></pre>
  
  <p>Creating empty matrix using DataFrame:</p>
  
  <pre><code>data_ibs = pd.DataFrame(index=table.columns,columns=table.columns)
  </code></pre>
  
  <p>Then doing calculation of pearsonr using for loops for the matrix:</p>
  
  <pre><code>for i in range(0,len(data_ibs.columns)) :
      for j in range(0,len(data_ibs.columns)) :
          data_ibs.ix[i,j] =  pearsonr(table.ix[:,i],table.ix[:,j]) [0]
  </code></pre>
  
  <p>It works but <em>waaay</em> to slow - it worked for 4 hours and still calculates. Is there another possibility to do that? </p>
  
  <p>Thanks in advance, I'll appreciate your ideas!</p>