text: user9466: My data look like this
  
  They are compound similarity estimations(the whole file is around 10GB). What I am trying to achieve is to compare the similarity distributions of each compound using the Kolmogorov-Smirnoff test so as to use the data for AIC calculation.
  This is my code:
  import pandas as pd
  from scipy.stats import ks_2samp
  import re
  
  with open('ecfp4_file.csv', 'r') as f, open('Metrics.tsv', 'a') as f_out:
      f_out.write('compound_1' + '\t' + 'compound_2' + '\t' + 'Similarity' + '\t' + 'KS Distance' + '\n')
      df = pd.read_csv(f, delimiter = ',', lineterminator = '\n', header = 0)
  
      for m in range(10, 99):
          dc1 = []
          dc2 = []
          m1 = m/100
          m2 = m/100+0.009
          df2 = df.loc[(df['ECFP4'] >= m1) & (df['ECFP4'] < m2)]          #Work on different thresholds each time
      
          for i in range(0, df2.shape[0]):
              if len(df2.index[df2['id1'] == df2.iloc[i, 0]].tolist()) >= 5:
                  dc1.append(df2.iloc[i, 0])
                  dc2.append(df2.iloc[i, 1])
          for i in range(0, len(dc1)):
              dc1_l = []
              dc2_l = []
              df3 = df.loc[df['id1'] == dc1[i]]
              dc1_l = df3.iloc[:, 2].tolist()
              df4 = df.loc[df['id1'] == dc2[i]]
              dc2_l = df4.iloc[:, 2].tolist()
              x1 = re.findall(r"statistic=(.*)\,.*$", str(ks_2samp(dc1_l, dc2_l)))
              f_out.write(str(dc1[i]) + '\t' + str(dc2[i]) + '\t' + str(m1) + '\t' + str(x1[0]) + '\n')
  
  which so far has taken 10 hours to run and it has not yet passed the 0.1 threshold, so it is highly unpractical. Does anyone know of a better way to do this? Or can help me refine my code for faster and better results?
  I also tried to refine it and run it in R, but I get a vector memory exhausted (limit reached) error when I run it
  install.packages("ggplot2")
  library(ggplot2)
  PS22_ECFP4 <- read.csv("./PS22_ECFP4.csv")
  
  df <- data.frame(matrix(ncol = 4, nrow = 0))
  cols <- c("id1", "id2", "Similarity", "KS Distance")
  colnames(df) <- cols
  
  for (i in PS22_ECFP4) {
    sim_df <- PS22_ECFP4[PS22_ECFP4$id1 == i,]
    dist1 <- sim_df[ , 3]
    for (k in 1:nrow(sim_df)) {
      sim <- sim_df[k , 3]
      sim_r <- floor(sim * 100) / 100
      comp2 <- sim_df[k, 2]
      comp2_df <- PS22_ECFP4[PS22_ECFP4$id1==comp2,]
      dist2 <- comp2_df[ , 3]
      if (length(dist1) > 1 & length(dist2) > 1) {
        ks <- ks.test(dist1, dist2)
        df[nrow(df) + 1,] = c(i, comp2, sim_r, ks$statistic)
      }
    }
  }
  
  write.csv(df, "./Metrics.csv", row.names = FALSE)
  
  
  user3263: I am not sure how useful this answer is. But may be you could try this.
  For your python script:
  You use f_out.write within a loop. I think usually this I/O operation is resource intensive. I guess the speed would improve significantly if you store the string in a variable and write the output only once outside the for loop like this:
  out_str = ''
  for i in range(0, len(dc1)):
              dc1_l = []
              dc2_l = []
              df3 = df.loc[df['id1'] == dc1[i]]
              dc1_l = df3.iloc[:, 2].tolist()
              df4 = df.loc[df['id1'] == dc2[i]]
              dc2_l = df4.iloc[:, 2].tolist()
              x1 = re.findall(r"statistic=(.*)\,.*$", str(ks_2samp(dc1_l, dc2_l)))
              out_str +=  (str(dc1[i]) + '\t' + str(dc2[i]) + '\t' + str(m1) + '\t' + str(x1[0]) + '\n')
  
  f_out.write(out_str)
  
  Regarding the memory error in your R script, are you using 32bit build of R?
  Please see this:
  http://web.mit.edu/~r/current/arch/i386_linux26/lib/R/library/base/html/Memory-limits.html#
  They say "Under most 64-bit versions of Windows the limit for a 32-bit build of R is 4Gb: for the oldest ones it is 2Gb. The limit for a 64-bit build of R (imposed by the OS) is 8Tb."
  Conclusion: Please try using your python script, but use the write operation outside the loop.
  
original_text: 9466: <p>My data look like this
  <a href="https://i.stack.imgur.com/nEDDO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nEDDO.png" alt="enter image description here" /></a></p>
  <p>They are compound similarity estimations(the whole file is around 10GB). What I am trying to achieve is to compare the similarity distributions of each compound using the Kolmogorov-Smirnoff test so as to use the data for AIC calculation.</p>
  <p>This is my code:</p>
  <pre><code>import pandas as pd
  from scipy.stats import ks_2samp
  import re
  
  with open('ecfp4_file.csv', 'r') as f, open('Metrics.tsv', 'a') as f_out:
      f_out.write('compound_1' + '\t' + 'compound_2' + '\t' + 'Similarity' + '\t' + 'KS Distance' + '\n')
      df = pd.read_csv(f, delimiter = ',', lineterminator = '\n', header = 0)
  
      for m in range(10, 99):
          dc1 = []
          dc2 = []
          m1 = m/100
          m2 = m/100+0.009
          df2 = df.loc[(df['ECFP4'] &gt;= m1) &amp; (df['ECFP4'] &lt; m2)]          #Work on different thresholds each time
      
          for i in range(0, df2.shape[0]):
              if len(df2.index[df2['id1'] == df2.iloc[i, 0]].tolist()) &gt;= 5:
                  dc1.append(df2.iloc[i, 0])
                  dc2.append(df2.iloc[i, 1])
          for i in range(0, len(dc1)):
              dc1_l = []
              dc2_l = []
              df3 = df.loc[df['id1'] == dc1[i]]
              dc1_l = df3.iloc[:, 2].tolist()
              df4 = df.loc[df['id1'] == dc2[i]]
              dc2_l = df4.iloc[:, 2].tolist()
              x1 = re.findall(r&quot;statistic=(.*)\,.*$&quot;, str(ks_2samp(dc1_l, dc2_l)))
              f_out.write(str(dc1[i]) + '\t' + str(dc2[i]) + '\t' + str(m1) + '\t' + str(x1[0]) + '\n')
  </code></pre>
  <p>which so far has taken 10 hours to run and it has not yet passed the 0.1 threshold, so it is highly unpractical. Does anyone know of a better way to do this? Or can help me refine my code for faster and better results?</p>
  <p>I also tried to refine it and run it in R, but I get a vector memory exhausted (limit reached) error when I run it</p>
  <pre><code>install.packages(&quot;ggplot2&quot;)
  library(ggplot2)
  PS22_ECFP4 &lt;- read.csv(&quot;./PS22_ECFP4.csv&quot;)
  
  df &lt;- data.frame(matrix(ncol = 4, nrow = 0))
  cols &lt;- c(&quot;id1&quot;, &quot;id2&quot;, &quot;Similarity&quot;, &quot;KS Distance&quot;)
  colnames(df) &lt;- cols
  
  for (i in PS22_ECFP4) {
    sim_df &lt;- PS22_ECFP4[PS22_ECFP4<span class="math-container">$id1 == i,]
    dist1 &lt;- sim_df[ , 3]
    for (k in 1:nrow(sim_df)) {
      sim &lt;- sim_df[k , 3]
      sim_r &lt;- floor(sim * 100) / 100
      comp2 &lt;- sim_df[k, 2]
      comp2_df &lt;- PS22_ECFP4[PS22_ECFP4$id1==comp2,]
      dist2 &lt;- comp2_df[ , 3]
      if (length(dist1) &gt; 1 &amp; length(dist2) &gt; 1) {
        ks &lt;- ks.test(dist1, dist2)
        df[nrow(df) + 1,] = c(i, comp2, sim_r, ks$</span>statistic)
      }
    }
  }
  
  write.csv(df, &quot;./Metrics.csv&quot;, row.names = FALSE)
  </code></pre>
   3263: <p>I am not sure how useful this answer is. But may be you could try this.</p>
  <p>For your python script:</p>
  <p>You use <code>f_out.write</code> within a loop. I think usually this I/O operation is resource intensive. I guess the speed would improve significantly if you store the string in a variable and write the output only once outside the for loop like this:</p>
  <pre><code>out_str = ''
  for i in range(0, len(dc1)):
              dc1_l = []
              dc2_l = []
              df3 = df.loc[df['id1'] == dc1[i]]
              dc1_l = df3.iloc[:, 2].tolist()
              df4 = df.loc[df['id1'] == dc2[i]]
              dc2_l = df4.iloc[:, 2].tolist()
              x1 = re.findall(r&quot;statistic=(.*)\,.*$&quot;, str(ks_2samp(dc1_l, dc2_l)))
              out_str +=  (str(dc1[i]) + '\t' + str(dc2[i]) + '\t' + str(m1) + '\t' + str(x1[0]) + '\n')
  
  f_out.write(out_str)
  </code></pre>
  <p>Regarding the memory error in your R script, are you using 32bit build of R?</p>
  <p>Please see this:</p>
  <p><a href="http://web.mit.edu/%7Er/current/arch/i386_linux26/lib/R/library/base/html/Memory-limits.html#" rel="nofollow noreferrer">http://web.mit.edu/~r/current/arch/i386_linux26/lib/R/library/base/html/Memory-limits.html#</a></p>
  <p>They say &quot;Under most 64-bit versions of Windows the limit for a 32-bit build of R is 4Gb: for the oldest ones it is 2Gb. The limit for a 64-bit build of R (imposed by the OS) is 8Tb.&quot;</p>
  <p><strong>Conclusion:</strong> Please try using your python script, but use the write operation outside the loop.</p>
  