title: django-celery infrastructure over multiple servers, broker is redis
body: <p>Currently we have everything setup on single cloud server, that includes:</p>
  
  <ul>
  <li>Database server</li>
  <li>Apache</li>
  <li>Celery</li>
  <li>redis to serve as a broker for celery and for some other tasks</li>
  <li>etc</li>
  </ul>
  
  <p>Now we are thinking to break apart the main components to separate servers e.g. separate database server, separate storage for media files, web servers behind load balancers. The reason is to not to pay for one heavy server and use load balancers to create servers on demand to reduce cost and improve overall speed.</p>
  
  <p>I am really confused about celery only, have anyone ever used celery on multiple production servers behind load balancers? Any guidance would be appreciated.</p>
  
  <p>Consider one small use case which is currently how it is been done on single server (confusion is that how that can be done when we use multiple servers):</p>
  
  <ul>
  <li>User uploads a <code>abc.pptx</code> file->reference is stored in database->stored on server disk</li>
  <li>A task (convert document to pdf) is created and goes in redis (broker) queue</li>
  <li>celery which is running on same server picks the task from queue
  <ul>
  <li>Read the file, convert it to pdf using software called <code>docsplit</code></li>
  <li>create a folder on server disk (which will be used as static content later on) puts pdf file and its thumbnail and plain text and the original file</li>
  </ul></li>
  </ul>
  
  <p>Considering the above use case, how can you setup up multiple web servers which can perform the same functionality?</p>