title: How to scale operations with a massive dictionary of lists in Python?
body: <p>I'm dealing with a "big data" problem in python, and I am really struggling for scalable solutions. </p>
  
  <p>The data structure I currently have is a massive dictionary of lists, with millions of keys and lists with millions of items. I need to do an operation on the items in the list. The problem is two-fold:</p>
  
  <p>(1) How to do scalable operations on a data structure this size?</p>
  
  <p>(2) How to do this with constraints of memory?</p>
  
  <p>For some code, here's a very basic example of a dictionary of lists:</p>
  
  <pre><code>example_dict1 = {'key1':[367, 30, 847, 482, 887, 654, 347, 504, 413, 821],
      'key2':[754, 915, 622, 149, 279, 192, 312, 203, 742, 846], 
      'key3':[586, 521, 470, 476, 693, 426, 746, 733, 528, 565]}
  </code></pre>
  
  <p>For a simple operation on the elements, the normal approach would be to iterate through the dictionary:</p>
  
  <pre><code>new_dictionary = {}
  
  for k, v in example_dict.items():
      new_list = []    
      for i in v:      ## iterate through dictionary lists
          i = compute_something(i) ## this is just an example, e.g. i**2 or i-13 would also work
          new_list.append(i)
      # now, create a new dictionary
      new_dictionary[k] = new_list
  </code></pre>
  
  <p>The problem is, this cannot work for a dictionary this size---I'm working on a sever with over 250GB of RAM, and it quickly becomes too large for memory. It's also too slow/not scalable, as that's a single iteration with one processor. </p>
  
  <p>Are there scalable solutions to this problem? </p>
  
  <p>Maybe it would work to somehow break up the dictionary, do the calculations with multiprocessing, and aggregate? Or is there a way to save this data to disk? </p>
  
  <p>I'm at a loss for ideas...</p>