title: Autoscale Python Celery with Amazon EC2
body: <p>I have a Celery Task-Manager to crunch some numbers for company analytics. </p>
  
  <p>The Task-Manager and workers are hosted on an Amazon EC2 Linux Server. </p>
  
  <p>I need to set up the system such if we send too many tasks to celery Amazon automatically sets up a new EC2 instance to run more workers and balances the load across these workers. </p>
  
  <p>The services that I'm aware exist are the Amazon Autoscale and Amazon Load balancing services which seem like exactly what I want to use however, I'm not sure what the best way to configure the Celery is.</p>
  
  <p>I <em>think</em> that I ought to have a celery "master" which is collecting all the tasks and a number of celery workers which execute them. As the number of tasks increases I want to add more workers. The way the autoscale works (by taking an AMI of the celery server) I think that I'm currently cloning the Master as well as the workers which seems like not what I want to do. </p>
  
  <p>How do I organise this to achieve my end goal which is flexible autoscaling task management using Celery to manage the tasks and Amazon Web Service to host the computing. </p>
  
  <p>As much detail as possible in any answers (or links to tutorials!) would be greatly appreciated as most tutorials or advice seems to assume large quantities of knowledge which I don't currently have!</p>