title: Reduce memory footprint from for loops and lists
body: <p>I have a function that I am trying to reduce the memory footprint of. The maximum amount of memory I can use is only 500MB. It seems that using <code>.split('\t')</code> and for loops is really using a lot of memory. Is there are way that I can reduce this memory usage?</p>
  
  <pre><code>Line #    Mem usage  Increment   Line Contents
  ==============================================
  10     35.4 MiB      0.0 MiB   @profile
  11                             def function(username):
  12     35.4 MiB      0.0 MiB       key = s3_bucket.get_key(username)
  13     85.7 MiB     50.2 MiB       file_data = key.get_contents_as_string()
  14    159.3 MiB     73.6 MiB       g = [x for x in file_data.splitlines() if not x.startswith('#')]
  15    144.8 MiB    -14.5 MiB       del file_data
  16    451.8 MiB    307.1 MiB       data = [x.split('\t') for x in g]
  17    384.0 MiB    -67.8 MiB       del g
  18
  19    384.0 MiB      0.0 MiB       d = []
  20    661.7 MiB    277.7 MiB       for row in data:
  21    661.7 MiB      0.0 MiB           d.append({'key': row[0], 'value':row[3]})
  22    583.7 MiB    -78.0 MiB       del data
  25    700.8 MiB    117.1 MiB       database[username].insert_many(d)
  26    700.8 MiB      0.0 MiB       return
  </code></pre>
  
  <p>UPDATE1</p>
  
  <p>As per the suggestion of @Jean-FrançoisFabre and @Torxed, it's an improvement but the generators still seem to take a large amount of memory.</p>
  
  <p>@martineau I'd prefer to use MongoDB <code>.insert_many()</code> as iterating over the keys and performing <code>.insert()</code> is <em>much</em> slower.</p>
  
  <pre><code>20     35.3 MiB      0.0 MiB   @profile
  21                             def function(username):
  22     85.4 MiB     50.1 MiB       file_data = s3_bucket.get_key(username).get_contents_as_string()
  23    610.5 MiB    525.2 MiB       data = (x.split('\t') for x in isplitlines(file_data) if not x.startswith('#'))
  24    610.5 MiB      0.0 MiB       d = ({'key': row[0], 'value':row[3]} for row in data)
  25    123.3 MiB   -487.2 MiB       database[username].insert_many(d)
  26    123.3 MiB      0.0 MiB       return
  </code></pre>
  
  <p>UDPATE2</p>
  
  <p>I've identified the source of the memory usage as this profile shows:</p>
  
  <pre><code>21     41.6 MiB      0.0 MiB   @profile
  22                             def insert_genotypes_into_mongodb(username):
  23     91.1 MiB     49.4 MiB       file_data = s3_bucket.get_key(username).get_contents_as_string()
  24     91.1 MiB      0.0 MiB       genotypes = (x for x in isplitlines(file_data) if not x.startswith('#'))
  25     91.1 MiB      0.0 MiB       d = ({'rsID': row.split('\t')[0], 'genotype':row.split('\t')[3]} for row in genotypes)
  26                                 # snps_database[username].insert_many(d)
  27     91.1 MiB      0.0 MiB       return
  </code></pre>
  
  <p>The <code>insert_many()</code> function clearly resolves the previous lines causing the whole list to be loaded into memory and confuses the profiler.</p>
  
  <p>The solution is insert the keys into MongoDB in chunks:</p>
  
  <pre><code>22     41.5 MiB      0.0 MiB   @profile
  23                             def insert_genotypes_into_mongodb(username):
  24     91.7 MiB     50.2 MiB       file_data = s3_bucket.get_key(username).get_contents_as_string()
  25    180.2 MiB     88.6 MiB       genotypes = (x for x in isplitlines(file_data) if not x.startswith('#'))
  26    180.2 MiB      0.0 MiB       d = ({'rsID': row.split('\t')[0], 'genotype':row.split('\t')[3]} for row in genotypes)
  27     91.7 MiB    -88.6 MiB       chunk_step = 100000
  28
  29     91.7 MiB      0.0 MiB       has_keys = True
  30    127.4 MiB     35.7 MiB       keys = list(itertools.islice(d,chunk_step))
  31    152.5 MiB     25.1 MiB       while has_keys:
  32    153.3 MiB      0.9 MiB           snps_database[username].insert_many(keys)
  33    152.5 MiB     -0.9 MiB           keys = list(itertools.islice(d,chunk_step))
  34    152.5 MiB      0.0 MiB           if len(keys) == 0:
  35    104.9 MiB    -47.6 MiB               has_keys = False
  36                                 # snps_database[username].insert_many(d[i*chunk_step:(i+1)*chunk_step])
  37    104.9 MiB      0.0 MiB       return
  </code></pre>
  
  <p>Thanks for all the help.</p>