title: Accelerate bulk insert using Django's ORM?
body: <p>I'm planning to upload a billion records taken from ~750 files (each ~250MB) to a db using django's ORM.
  Currently each file takes ~20min to process, and I was wondering if there's any way to accelerate this process.</p>
  
  <p>I've taken the following measures:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/1136106/efficent-way-to-insert-thousands-of-records-into-a-table-sqlite-python-django">Use @transaction.commit_manually</a> and commit once every 5000 records</li>
  <li>Set DEBUG=False so that django <a href="https://stackoverflow.com/questions/4292963/python-process-keeps-growing-in-django-db-upload-script">won't accumulate all the sql commands in memory</a></li>
  <li>The loop that runs over records in a single file is completely contained in a single function (minimize stack changes)</li>
  <li>Refrained from hitting the db for queries (used a local hash of objects already in the db <a href="https://stackoverflow.com/questions/2252530/efficent-way-to-bulk-insert-with-get-or-create-in-django-sql-python-django">instead of using get_or_create</a>)</li>
  <li><a href="http://docs.djangoproject.com/en/dev/ref/models/instances/#forcing-an-insert-or-update" rel="noreferrer">Set force_insert=True in the save()</a> in hopes it will save django some logic</li>
  <li><a href="http://docs.djangoproject.com/en/dev/ref/models/instances/#explicitly-specifying-auto-primary-key-values" rel="noreferrer">Explicitly set the id</a> in hopes it will save django some logic</li>
  <li>General code minimization and optimization</li>
  </ul>
  
  <p>What else can I do to speed things up? Here are some of my thoughts:</p>
  
  <ul>
  <li>Use some kind of Python compiler or version which is quicker (Psyco?)</li>
  <li><a href="https://stackoverflow.com/questions/2252530/efficent-way-to-bulk-insert-with-get-or-create-in-django-sql-python-django">Override the ORM and use SQL directly</a></li>
  <li>Use some 3rd party code that might be better (<a href="http://code.google.com/p/django-batchimport/" rel="noreferrer">1</a>, <a href="http://djangosnippets.org/snippets/446/" rel="noreferrer">2</a>)</li>
  <li><a href="http://code.djangoproject.com/ticket/661" rel="noreferrer">Beg the django community</a> to create a bulk_insert function</li>
  </ul>
  
  <p>Any pointers regarding these items or any other idea would be welcome :)</p>