title: Chaining or chording very large job groups/tasks with Celery & Redis
body: <p>I'm working on a project to parallelize some heavy simulation jobs. Each run takes about two minutes, takes 100% of the available CPU power, and generates over 100 MB of data. In order to execute the next step of the simulation, those results need to be combined into one huge result.</p>
  <p>Note that this will be run on performant systems (currently testing on a machine with 16 GB ram and 12 cores, but will probably upgrade to bigger HW)</p>
  <p>I can use a celery job group to easily dispatch about 10 of these jobs, and then chain that into the concatenation step and the next simulation. (Essentially a Celery chord) However, I need to be able to run at least 20 on this machine, and eventually 40 on a beefier machine. It seems that Redis doesn't allow for large enough objects on the result backend for me to do anything more than 13. I can't find any way to change this behavior.</p>
  <p>I am currently doing the following, and it works fine:</p>
  <pre><code>test_a_group = celery.group(test_a(x) for x in ['foo', 'bar'])
  
  test_a_result = rev_group.apply_async(add_to_parent=False)
  
  return = test_b(test_a_result.get())
  </code></pre>
  <p>What I would rather do:</p>
  <pre><code>return chord(test_a_group, test_b())
  </code></pre>
  <p>The second one works for small datasets, but not large ones. It gives me a non-verbose 'Celery ChordError 104: connection refused' with large data.</p>
  <p><strong>Test B returns very small data, essentially a pass fail, and I am only passing the group result into B, so it should work, except that I think the entire group is being appended to the result of B, in the form of parent, making it too big. I can't find out how to prevent this from happening.</strong></p>
  <p>The first one works great, and I would be okay, except that it complains, saying:</p>
  <blockquote>
  <p>[2015-01-04 11:46:58,841: WARNING/Worker-6] /home/cmaclachlan/uriel-venv/local/lib/python2.7/site-packages/celery/result.py:45:</p>
  <p>RuntimeWarning: Never call result.get() within a task!
  See <a href="http://docs.celeryq.org/en/latest/userguide/tasks.html#task-synchronous-subtasks" rel="nofollow noreferrer">http://docs.celeryq.org/en/latest/userguide/tasks.html#task-synchronous-subtasks</a></p>
  <p>In Celery 3.2 this will result in an exception being
  raised instead of just being a warning.</p>
  <p>warnings.warn(RuntimeWarning(E_WOULDBLOCK))</p>
  </blockquote>
  <p>What the link essentially suggests is what I want to do, but can't.</p>
  <p>I think I read somewhere that Redis has a limit of 500 mb on size of data pushed to it.</p>
  <p>Any advice on this hairiest of problems?</p>