title: How should I implement callback for taskset in celery
body: <h1>Question</h1>
  
  <p>I use celery to launch task sets that look like this: </p>
  
  <ol>
  <li>I perform a batch of tasks that can be run in parallel, number of tasks in this batch varies from tens to couple thousands. </li>
  <li><p>I aggregate results of these tasks into single answer, then do something with this answer --- like store to the database, save to special result file and so on. Basically after tasks done executing I have to call function that has following signature: </p>
  
  <pre><code>def callback(result_file_name, task_result_list): 
      #store in file
  
  
  def callback(entity_key, task_result_list):
      #store in db 
  </code></pre></li>
  </ol>
  
  <p>For now step 1. is done in Celery queue and step 2 is done outside celery: </p>
  
  <pre><code>    tasks = []
  
      # add taksks to tasks list 
  
      task_group = group()
      task_group.tasks = tasks
  
      result = task_group.apply_async()
  
      res = result.join()
  
      # Aggregate results 
  
      # Save results to file, database whatever
  </code></pre>
  
  <p>This approach is cumbersome since I have to stop a single thread until all tasks are performed (which can take couple of hours). </p>
  
  <p>I would like to somehow move step 2 to celery also --- esentially I would need to add a callback to entire taskset (as far as I know it is unsupported in Celery) or submit a task that is executed after all these subtasks. </p>
  
  <p>Does anyone have idea how to do it? I use it in the django enviorment so I can store some state in the database.</p>
  
  <h1>To sum up my recent findings</h1>
  
  <h2>Chords won't do</h2>
  
  <p>I'cant use chords straight forwardly because chords enable me to create callbacks that look this way: </p>
  
  <pre><code>    def callback(task_result_list): 
          #store in file
  </code></pre>
  
  <p>there is no obvious way to pass additional parameters to callback (especially because these callbacks can't be local functions). </p>
  
  <h2>Using the database either</h2>
  
  <p>I can store results using <code>TaskSetMeta</code> but this entity has no status field --- so even if I would add a signal to TaskSetMeta i'd have to pool task results which could have siginificant overhead. </p>