title: How to implement a multiprocessing pool using Celery
body: <p>In python multiprocessing, am able to create a multiprocessing pool of say 30 processes to process some long running equation on some IDs. The below code spawns 30 processes on an 8 core machine and the <code>load_average</code> never exceeds 2.0. In fact, the 30 consumers is a limit given that the server where the postgresql database that hosts the IDs has got 32 cores so I know I can spawn more processes if my database could handle it.</p>
  
  <pre><code>from multiprocessing import Pool
  number_of_consumers = 30
  pool = Pool(number_of_consumers)
  </code></pre>
  
  <p>I have taken the time to setup Celery but am unable to recreate the 30 processes. I thought setting the concurrency e.g. <code>-c 30</code> would create 30 processes but if am not wrong that means I have 32 processors which I intend to use which is wrong as I only have 8! Also, am seeing the <code>load_average</code> hitting 10.0 on an 8 core machine which is bad..</p>
  
  <pre><code>[program:my_app]
  command = /opt/apps/venv/my_app/bin/celery -A celery_conf.celeryapp worker -Q app_queue -n app_worker --concurrency=30 -l info
  </code></pre>
  
  <p>So, when using Celery, how can I recreate my 30 processes on a 8 core machine?</p>
  
  <p><strong>Edit: Qualifying the Confusion</strong></p>
  
  <p>I thought I'd attach an image to illustrate my confusion on server load when discussing Celery and Python Multiprocessing. The server am using has 8 cores. Using Python Multiprocessing and spawning 30 processes, the load average as seen in the attached diagram is at 0.22 meaning <em>-if my linux knowledge serves me right-</em> that my script is using one core to spawn the 30 processes hence a very low load_average.</p>
  
  <p><img src="https://i.stack.imgur.com/suumm.png" alt="load_average using python multiprocessing screenshot"></p>
  
  <p>My understanding of the <code>--concurrency=30</code> option in celery is that it instructs Celery how many <strong>cores</strong> it will use rather than how many processes it is required to spawn. Am I right on that? Is there a way to instruct Celery to use 2 cores and for each core spawn 15 processes giving me a total of 30 concurrent processes so that my server load remains low?</p>