text: user3848: Currently algorithmically developing a calculation for "co-dependent" mutation (quotation appropriate). The algorithm uses chunked biopython alignment objects which are parallelized.
  Aim
  One optization is caching a database within parallelization, but I ain't sure whether the parallelization duplicates the cache.
  Method
  Each alignment chunk (biopython object) is passed onto a separate CPU and refers back to the original database to obtain its metadata in the final stage of the calculation (dragging metadata around this algorithm makes no sense).
  The idea is to cache the database to prevent pulling it in from the harddrive:
  from multiprocessor import Pool
  from myOOPlibrary import mystuff    
  
  Facade design, the parallelised index to the alignment chunks enter into the myOOPlibrary:
  from cachetools import cached, TTLCache
  
  cache = TTLCache(maxsize=1, ttl=500)
  
  Class mystuff (stuff2):
  ... # init omitted
  
      def mymethod(self, ..):
      # to be updated 
      
  Class stuff2 ():
  ... # init omitted
  
      @cached(cache)
      def getDB(self, ...):
  .... #  pull it in (harddrive) and parse to a dictionary
      return metadataDB
  
  Question
  I'm wondering if caching is the right approach in parallelization, specifically whether this will end up with multiplying the database held in RAM...
  Alternatives
  The alternativees to caching are:
  
  IO to JSON (not cool)
  IO via pickle (binary IO): a JSON will readilypickle
  
  The possible advantage of the alternativees are the database is written once and accessed by all cores for certain.
  Underlying question
  
  When caching is the database  cached for each core? If so the RAM overhead becomes an issue because the database is duplicated total CPUs - 1 times
  Alternatively is the cache shared between CPUs centrally and each CPU refers to this location?
  
  
  user3848: Summarising
  
  @Jesse's comment I think is the right answer, open the database outside parallelisation. What I didn't thoroughly describe (and what was essentially requested) is the architecture, the alignments entering into an entire OOP library and the whole library is parallelised, i.e. every new alignment entering library is on its own CPU. The answer is obvious: instantiate the OOP library with the entire database and thats a key purpose of the design.
  The technical answer is joblib. This is built to cache across parallelised archetechture as the joblib team point out here. The criticism is that there's parallelisation within parallelisation. The discussion* is as much in the comments as in the response.
  @user172818 response is correct too, the algorithm would benefit from C. Its a lot of code to rewrite in C, but sidestepping via Cython is a future development (its a hybrid language connecting C and Python).
  
  *, https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache
  
  There was a comment hinting this is Stack Overflow, that's difficult. Ultimately the technical answer (which is Stack Overflow) isn't the correct answer as far as I understand OOP.
  
  Feedback
  Just to wrap this, point 1. worked fine, thank you. It means a database sits on each processor in parallelisation, but is imported  once. In truth, it worked nicely and provided cleaner code. Re-writing to instantiate once would affect other stuff so would be more costly than a cost-saving.
  Python can be very fast (e.g. numpy), but also really slow (hence it has critiques), extensively parallelising helps bridge the gap.
  Python parallelisation has critiques which I strongly, strongly believe are unfounded: I parallelise loads via multiprocessor library no problems.
  Caching here, i.e. parallelising across a parallelisation could stretch things too far.
  Last point
  Point 3, re-writing in Cython/C, is a greater priority than the testing required for parallelised caching: can't parallelise all data and that code is better as C.
  
original_text: 3848: <p>Currently algorithmically developing a calculation for &quot;co-dependent&quot; mutation (quotation appropriate). The algorithm uses chunked biopython alignment objects which are parallelized.</p>
  <p><strong>Aim</strong>
  One optization is caching a database within parallelization, but I ain't sure whether the parallelization duplicates the cache.</p>
  <p><strong>Method</strong>
  Each alignment chunk (biopython object) is passed onto a separate CPU and refers back to the original database to obtain its metadata in the final stage of the calculation (dragging metadata around this algorithm makes no sense).</p>
  <p>The idea is to cache the database to prevent pulling it in from the harddrive:</p>
  <pre><code>from multiprocessor import Pool
  from myOOPlibrary import mystuff    
  </code></pre>
  <p>Facade design, the parallelised index to the alignment chunks enter into the myOOPlibrary:</p>
  <pre><code>from cachetools import cached, TTLCache
  
  cache = TTLCache(maxsize=1, ttl=500)
  
  Class mystuff (stuff2):
  ... # init omitted
  
      def mymethod(self, ..):
      # to be updated 
      
  Class stuff2 ():
  ... # init omitted
  
      @cached(cache)
      def getDB(self, ...):
  .... #  pull it in (harddrive) and parse to a dictionary
      return metadataDB
  </code></pre>
  <p><strong>Question</strong>
  I'm wondering if caching is the right approach in parallelization, specifically whether this will end up with multiplying the database held in RAM...</p>
  <p><strong>Alternatives</strong>
  The alternativees to caching are:</p>
  <ul>
  <li>IO to JSON (not cool)</li>
  <li>IO via pickle (binary IO): a JSON will readily<code>pickle</code></li>
  </ul>
  <p>The possible advantage of the alternativees are the database is written once and accessed by all cores for certain.</p>
  <p><strong>Underlying question</strong></p>
  <ul>
  <li>When caching is the database  cached for each core? If so the RAM overhead becomes an issue because the database is duplicated <code>total CPUs - 1</code> times</li>
  <li>Alternatively is the cache shared between CPUs centrally and each CPU refers to this location?</li>
  </ul>
   3848: <p><em><strong>Summarising</strong></em></p>
  <ol>
  <li>@Jesse's comment I think is the right answer, open the database outside parallelisation. What I didn't thoroughly describe (and what was essentially requested) is the architecture, the alignments entering into an entire OOP library and the whole library is parallelised, i.e. every new alignment entering library is on its own CPU. The answer is obvious: instantiate the OOP library with the entire database and thats a key purpose of the design.</li>
  <li>The technical answer is <code>joblib</code>. This is built to cache across parallelised archetechture as the <code>joblib</code> team point out <strong><a href="https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache">here</a></strong>. The criticism is that there's parallelisation within parallelisation. The discussion* is as much in the comments as in the response.</li>
  <li>@user172818 response is correct too, the algorithm would benefit from C. Its a lot of code to rewrite in <code>C</code>, but sidestepping via Cython is a future development (its a hybrid language connecting C and Python).</li>
  </ol>
  <p>*, <a href="https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache">https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache</a></p>
  <hr />
  <p>There was a comment hinting this is Stack Overflow, that's difficult. Ultimately the technical answer (which is Stack Overflow) isn't the correct answer as far as I understand OOP.</p>
  <hr />
  <p><em><strong>Feedback</strong></em></p>
  <p>Just to wrap this, point 1. worked fine, thank you. It means a database sits on each processor in parallelisation, but is imported  once. In truth, it worked nicely and provided cleaner code. Re-writing to instantiate once would affect other stuff so would be more costly than a cost-saving.</p>
  <p>Python can be very fast (e.g. numpy), but also really slow (hence it has critiques), extensively parallelising helps bridge the gap.
  Python parallelisation has critiques which I strongly, strongly believe are unfounded: I parallelise loads via <code>multiprocessor</code> library no problems.</p>
  <p>Caching here, i.e. parallelising across a parallelisation could stretch things too far.</p>
  <p><em><strong>Last point</strong></em></p>
  <p>Point 3, re-writing in Cython/<code>C</code>, is a greater priority than the testing required for parallelised caching: can't parallelise all data and that code is better as <code>C</code>.</p>
  