title: Efficient handling of long running HTTP connections in an nginx/gunicorn/django web architecture
body: <p>I am working on a web service implemented on top of <a href="http://nginx.org/" rel="noreferrer">nginx</a>+<a href="http://gunicorn.org/" rel="noreferrer">gunicorn</a>+<a href="https://www.djangoproject.com/" rel="noreferrer">django</a>.  The clients are smartphone applications.  The application needs to make some long running calls to external APIs (Facebook, Amazon S3...), so the server simply queues the job to a job server (using <a href="http://www.celeryproject.org/" rel="noreferrer">Celery</a> over <a href="http://redis.io/" rel="noreferrer">Redis</a>).</p>
  
  <p>Whenever possible, once the server has queued the job, it returns right away, and the HTTP connection is closed.  This works fine and allows the server to sustain very high load.</p>
  
  <pre><code>client                   server                 job server
    .                        |                        |
    .                        |                        |
    |------HTTP request-----&gt;|                        |
    |                        |--------queue job------&gt;|
    |&lt;--------close----------|                        |
    .                        |                        |
    .                        |                        |
  </code></pre>
  
  <p>But in some cases, the client needs to get the result as soon as the job is finished.  Unfortunately, there's no way the server can contact the client once the HTTP connection is closed.  One solution would be to rely on the client application polling the server every few seconds until the job is completed.  I would like to avoid this solution, if possible, mostly because it would hinder the reactiveness of the service, and also because it would load the server with many unnecessary poll requests.</p>
  
  <p>In short, I would like to keep the HTTP connection up and running, doing nothing (except perhaps sending a whitespace every once in a while to keep the TCP connection alive, just <a href="http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadComplete.html" rel="noreferrer">like Amazon S3 does</a>), until the job is done, and the server returns the result.</p>
  
  <pre><code>client                   server                 job server
    .                        |                        |
    .                        |                        |
    |------HTTP request-----&gt;|                        |
    |                        |--------queue job------&gt;|
    |&lt;------keep-alive-------|                        |
    |         [...]          |                        |
    |&lt;------keep-alive-------|                        |
    |                        |&lt;--------result---------|
    |&lt;----result + close-----|                        |
    .                        |                        |
    .                        |                        |
  </code></pre>
  
  <p><strong>How can I implement long running HTTP connections in an efficient way, assuming the server is under very high load (it is not the case yet, but the goal to be able to sustain the highest possible load, with hundreds or thousands of requests per second)?</strong></p>
  
  <p>Offloading the actual jobs to other servers should ensure a low CPU usage on the server, but how can I avoid processes piling up and using all the server's RAM, or incoming requests being dropped because of too many open connections?</p>
  
  <p>This is probably mostly a matter of configuring nginx and gunicorn properly.  I have read a bit about <a href="http://gunicorn.org/design.html#async-workers" rel="noreferrer">async workers based on greenlets in gunicorn</a>: the documentation says that async workers are used by "<em>Applications making long blocking calls (Ie, external web services)</em>", this sounds perfect.  It also says "<em>In general, an application should be able to make use of these worker classes with no changes</em>".  This sounds great.  Any feedback on this?</p>
  
  <p>Thanks for your advices.</p>