title: Finding intersections of huge sets with huge dicts
body: <p>I have a dict with 50,000,000 keys (strings) mapped to a count of that key (which is a subset of one with billions).</p>
  
  <p>I also have a series of objects with a class set member containing a few thousand strings that may or may not be in the dict keys.</p>
  
  <p>I need the fastest way to find the intersection of each of these sets.</p>
  
  <p>Right now, I do it like this code snippet below:</p>
  
  <pre><code>for block in self.blocks:
      #a block is a python object containing the set in the thousands range
      #block.get_kmers() returns the set
      count = sum([kmerCounts[x] for x in block.get_kmers().intersection(kmerCounts)])
      #kmerCounts is the dict mapping millions of strings to ints
  </code></pre>
  
  <p>From my tests so far, this takes about 15 seconds per iteration. Since I have around 20,000 of these blocks, I am looking at half a week just to do this. And that is for the 50,000,000 items, not the billions I need to handle...</p>
  
  <p>(And yes I should probably do this in another language, but I also need it done fast and I am not very good at non-python languages).</p>