title: How to iterate a large table in Django without running out of memory?
body: <p>I have a Django model whose table has millions of records in it.  I'm trying to do some emergency maintenance on all of the records in the table at a shell but I'm unable to do a <code>MyModel.objects.all()</code> without completely exhausting memory on my system.</p>
  
  <p>Even a <code>pass</code> causes the OOM killer to be called, killing my process:</p>
  
  <pre><code>for ii in MyModel.objects.all():
      pass
  </code></pre>
  
  <p>The reason is because Django's <code>QuerySet</code> is trying to build up its "result cache", by building a list with <em>all</em> of my records in it, here:</p>
  
  <pre><code># django/db/models/query.py
  def _fetch_all(self):
      if self._result_cache is None:
          self._result_cache = list(self.iterator())  # &lt;&lt;&lt;&lt; this guy!
      if self._prefetch_related_lookups and not self._prefetch_done:
          self._prefetch_related_objects()
  </code></pre>
  
  <p>But my machine can't hold the whole list in memory.</p>
  
  <p>Of course, iterating <code>.all()</code> on such a large table would be a terrible idea in a real app, so the scope of this problem is rather limited (maintenance activities) but it does come up from time to time.</p>