title: Celery - Chaining remote callbacks
body: <p>I have three Celery tasks that run on three different servers respectively.</p>
  
  <ul>
  <li>tasks.send_push_notification</li>
  <li>tasks.send_sms</li>
  <li>tasks.send_email</li>
  </ul>
  
  <p>I want to setup a workflow such that if sending push notification fails, I should try sending sms. And if sending sms fails, I should send email.</p>
  
  <p>If those 3 tasks and their code base was on the same server, I would have followed <a href="http://docs.celeryproject.org/en/latest/userguide/canvas.html#the-primitives" rel="nofollow">the example on chained tasks</a> and done something like</p>
  
  <pre><code>from celery import chain
  from tasks import send_push_notification, send_sms, send_email
  import json
  # some paylaod
  payload = json.dumps({})
  res = chain(
      send_push_notification.subtask(payload),
      send_sms.subtask(payload),
      send_email.subtask(payload)
  )()
  </code></pre>
  
  <p>But the tasks are kept on 3 different servers!</p>
  
  <p>I have tried</p>
  
  <pre><code># 1
  from celery import chain
  from my_celery_app import app
  res = chain(
      app.send_task('tasks.send_push_notification', payload),
      app.send_task('tasks.send_sms', payload),
      app.send_task('tasks.send_email', payload)
  )()
  # Which fails because I am chaining tasks not subtasks
  </code></pre>
  
  <p>and</p>
  
  <pre><code># 2
  from celery import chain, subtask
  res = chain(
      subtask('tasks.send_push_notification', payload), 
      subtask('tasks.send_sms', payload), 
      subtask('tasks.send_email', payload)
  )()
  # fails because I am not adding the tasks on the broker
  </code></pre>
  
  <p>How can this be done?</p>
  
  <p><strong>Update:</strong>
  I can do it using <code>link</code> NOT <code>chain</code>.</p>
  
  <pre><code>from celery import subtask
  res = app.send_task(
      'tasks.send_push_notification', (payload, ),
      link=subtask(
          'tasks.send_sms', (payload, ),
          link=subtask(
              'tasks.send_email', (payload, ),
          )
      )
  )
  </code></pre>
  
  <p>There is a lot of nesting. And because I actually need to create a database driven workflow, it will be complicated to create it this way.</p>