text: user57: I have a script that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).
  The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.
  I don't really get it; the script actually prints the last line 
  
  *_families_2.tsv and *_coverages_2.tsv files saved.
  
  , but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed here, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).
  Does anybody an idea where the problem could be? Or how to figure out?
  The reported problem on GitHub.
  
  user3848: You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.
  The solution is to either:
  
  
  find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.
  
  parallelise the calculation (an excellent way to overcome RAM problems), or,
  
  find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.
  
  
  
  
original_text: 57: <p>I have <a href="https://github.com/tbenavi1/smudgeplot/blob/master/hetkmers.py" rel="nofollow noreferrer">a script</a> that extracts the set of unique kmer pairs from a kmer dump file (&amp; then using them downstream to guess ploidy).</p>
  
  <p>The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.</p>
  
  <p>I don't really get it; the script actually prints the last line </p>
  
  <blockquote>
    <p>*_families_2.tsv and *_coverages_2.tsv files saved.</p>
  </blockquote>
  
  <p>, but then it runs further (&amp; forever). I thought it could be related to longer time to free the memory as discussed <a href="https://stackoverflow.com/questions/2412715/python-program-doesnt-quit-when-finished">here</a>, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).</p>
  
  <p>Does anybody an idea where the problem could be? Or how to figure out?</p>
  
  <p>The reported problem on <a href="https://github.com/tbenavi1/smudgeplot/issues/18" rel="nofollow noreferrer">GitHub</a>.</p>
   3848: <p>You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.</p>
  <p>The solution is to either:</p>
  <blockquote>
  <ol>
  <li><p>find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.</p>
  </li>
  <li><p>parallelise the calculation (an excellent way to overcome RAM problems), or,</p>
  </li>
  <li><p>find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.</p>
  </li>
  </ol>
  </blockquote>
  