title: Optimizing Celery for third party HTTP calls
body: <p>We are using celery to make third party http calls. We have around 100+ of tasks which simply calls the third party HTTP API calls. Some tasks call the API's in bulk, for example half a million requests at 4 AM in morning, while some are continuous stream of API calls receiving requests almost once or twice per second.</p>
  
  <p>Most of API call response time is between 500 - 800 ms.</p>
  
  <p>We are seeing very slow delivery rates with celery. For most of the above tasks, the max delivery rate is around 100/s (max) to almost 1/s (min). I believe this is very poor and something is definitely wrong, but I am not able to figure out what it is.</p>
  
  <p>We started with cluster of 3 servers and incrementally made it a cluster of 7 servers, but with no improvement. We have tried with different concurrency settings from autoscale to fixed 10, 20, 50, 100 workers. There is no result backend and our broker is RabbitMQ.</p>
  
  <p>Since our task execution time is very small, less than a second for most, we have also tried making prefetch count unlimited to various values.</p>
  
  <p><code>--time-limit=1800 --maxtasksperchild=1000 -Ofair -c 64 --config=celeryconfig_production</code></p>
  
  <p>Servers are 64 G RAM, Centos 6.6.</p>
  
  <p>Can you give me idea on what could be wrong or pointers on how to solve it? </p>
  
  <p>Should we go with gevents? Though I have little of idea of what it is.</p>