title: Problems with Celery & Redis Backend
body: <p>I have a system set up currently that is using celery with a redis 
  backend to do a bunch of asynchronous tasks such as sending emails, 
  pulling social data, crawling,etc. Everything is working great, but I 
  am having group figuring out how to monitor the system (aka the number 
  of queue up messages). I started looking through the celery source but 
  I figured I would post my questions in here: 
  First off, here are my configurations: </p>
  
  <pre><code>BROKER_BACKEND                  = "redis" 
  BROKER_HOST                     = "localhost" 
  BROKER_PORT                     = 6379 
  BROKER_VHOST                    = "1" 
  REDIS_CONNECT_RETRY     = True 
  REDIS_HOST                              = "localhost" 
  REDIS_PORT                              = 6379 
  REDIS_DB                                = "0" 
  CELERY_SEND_EVENTS                      = True 
  CELERYD_LOG_LEVEL               = 'INFO' 
  CELERY_RESULT_BACKEND           = "redis" 
  CELERY_TASK_RESULT_EXPIRES      = 25 
  CELERYD_CONCURRENCY             = 8 
  CELERYD_MAX_TASKS_PER_CHILD = 10 
  CELERY_ALWAYS_EAGER                     =True
  </code></pre>
  
  <hr>
  
  <p>The first thing I am trying to do is monitor how many messages are in 
  my queue. I assume, behind the scenes, the redis backend is just 
  pushing/popping from a list, although I cannot seem to find that in 
  the code. So I mock up a simulation where I start about 100 tasks and 
  am trying to find them in redis: 
  My celeryd is running like this: 
  python manage.py celeryd -c 4 --loglevel=DEBUG -n XXXXX --logfile=logs/ 
  celery.log 
  So I should only have 4 concurrent workers at once ..... 
  Two thing I do not understand: 
  Problem 1: 
  After I have queued up 100 task, and look for them on redis, I only 
  see the following: </p>
  
  <pre><code>$ redis-cli 
  redis 127.0.0.1:6379&gt; keys * 
  1) "_kombu.binding.celery" 
  redis 127.0.0.1:6379&gt; select 1 
  OK 
  redis 127.0.0.1:6379[1]&gt; keys * 
  1) "_kombu.binding.celery" 
  2) "_kombu.binding.celeryd.pidbox" 
  redis 127.0.0.1:6379[1]&gt;
  </code></pre>
  
  <p>I cannot seem to find the tasks to get a number of how many are queued 
  (technically, 96 should be since I only support 4 concurrent tasks) </p>
  
  <p>Problem 2 </p>
  
  <pre><code>$ ps aux | grep celeryd | cut -c 13-120 
   41258   0.2  0.2  2526232   9440 s004  S+    2:27PM   0:07.35 python 
  manage.py celeryd -c 4 --loglevel=DEBU 
   41261   0.0  0.1  2458320   2468 s004  S+    2:27PM   0:00.09 python 
  manage.py celeryd -c 4 --loglevel=DEBU 
   38457   0.0  0.8  2559848  34672 s004  T    12:34PM   0:18.59 python 
  manage.py celeryd -c 4 --loglevel=INFO 
   38449   0.0  0.9  2517244  36752 s004  T    12:34PM   0:35.72 python 
  manage.py celeryd -c 4 --loglevel=INFO 
   38443   0.0  0.2  2524136   6456 s004  T    12:34PM   0:10.15 python 
  manage.py celeryd -c 4 --loglevel=INFO 
   84542   0.0  0.0  2460112      4 s000  T    27Jan12   0:00.74 python 
  manage.py celeryd -c 4 --loglevel=INFO 
   84536   0.0  0.0  2506728      4 s000  T    27Jan12   0:00.51 python 
  manage.py celeryd -c 4 --loglevel=INFO 
   41485   0.0  0.0  2435120    564 s000  S+    2:54PM   0:00.00 grep 
  celeryd 
   41264   0.0  0.1  2458320   2480 s004  S+    2:27PM   0:00.09 python 
  manage.py celeryd -c 4 --loglevel=DEBU 
   41263   0.0  0.1  2458320   2480 s004  S+    2:27PM   0:00.09 python 
  manage.py celeryd -c 4 --loglevel=DEBU 
   41262   0.0  0.1  2458320   2480 s004  S+    2:27PM   0:00.09 python 
  manage.py celeryd -c 4 --loglevel=DEBU 
  </code></pre>
  
  <p>If anyone could explain this for me, it would be great. </p>