title: Run task of same dag on different servers
body: <p>I need to run the following dag - </p>
  
  <pre><code>dag = DAG('dummy_for_testing', 
  default_args=default_args,schedule_interval=None)
  
  t1 = BashOperator(
      task_id='print_date',
      bash_command='date',
      dag=dag)
  
  t2 = BashOperator(
      task_id='print_host',
      bash_command='hostname',
      queue='druid_queue',
      dag=dag)
  
  t3 = BashOperator(
      task_id='print_directory',
      bash_command='pwd',
      dag=dag)
  
  t3.set_upstream(t2)
  t2.set_upstream(t1)
  </code></pre>
  
  <p>where t1 and t3 runs on server A and t2 runs on server B(queue=druid_queue).
  I am currently using <a href="https://github.com/puckel/docker-airflow" rel="nofollow noreferrer">puckel/docker-airflow</a> to set up airflow. The yml files for the server looks like :<br>
  Server1   </p>
  
  <pre><code>version: '2.1'
  services:
      redis:
          image: 'redis:3.2.7'
          ports:
              - "10.0.11.4:6999:6379"
          command: redis-server
  
      postgres:
          image: postgres:9.6
          container_name: postgres-airflow
          ports:
              - "10.0.11.4:5434:5432"
          environment:
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
  
      webserver:
          image: puckel/docker-airflow:1.10.2
          container_name: airflow
          restart: always
          depends_on:
              - postgres
              - redis
          environment:
              - LOAD_EX=n
              - FERNET_KEY=&lt;&gt;
              - EXECUTOR=Celery
              - user_logs_config_loc=dags/user_logs/configurations/
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
          volumes:
              - /data/druid-data/airflow/dags:/usr/local/airflow/dags
              - /var/run/docker.sock:/var/run/docker.sock
          ports:
              - "10.0.11.4:8085:8080"
          command: webserver
          healthcheck:
              test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
              interval: 30s
  
      flower:
          image: puckel/docker-airflow:1.10.2
          restart: always
          depends_on:
              - redis
          environment:
              - EXECUTOR=Celery
          ports:
              - "5555:5555"
          command: flower
  
      scheduler:
          image: puckel/docker-airflow:1.10.2
          restart: always
          depends_on:
              - webserver
          volumes:
              - /data/druid-data/airflow/dags:/usr/local/airflow/dags
          environment:
              - LOAD_EX=n
              - FERNET_KEY=&lt;&gt;
              - EXECUTOR=Celery
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
              - user_logs_config_loc=dags/user_logs/configurations/
          command: scheduler
  
      worker:
          image: puckel/docker-airflow:1.10.2
          restart: always
          depends_on:
              - scheduler
          volumes:
              - /data/druid-data/airflow/dags:/usr/local/airflow/dags
          environment:
              - FERNET_KEY=&lt;&gt;
              - EXECUTOR=Celery
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
              - user_logs_config_loc=dags/user_logs/configurations/
          command: worker
  </code></pre>
  
  <p>Server2</p>
  
  <pre><code>version: '2.1'
  services:
      redis:
          image: 'redis:3.2.7'
          ports:
              - "10.0.11.5:6999:6379"
          command: redis-server
  
      postgres:
          image: postgres:9.6
          container_name: postgres-airflow
          ports:
              - "10.0.11.5:5434:5432"
          environment:
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
  
      webserver:
          image: puckel/docker-airflow:latest
          container_name: airflow
          restart: always
          depends_on:
              - postgres
              - redis
          environment:
              - LOAD_EX=n
              - FERNET_KEY=&lt;&gt;
              - EXECUTOR=Celery
              - user_logs_config_loc=dags/user_logs/configurations/
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
          volumes:
              - /data/qa/druid-data/airflow/dags:/usr/local/airflow/dags
              - /var/run/docker.sock:/var/run/docker.sock
          ports:
              - "10.0.11.5:8085:8080"
          command: webserver
          healthcheck:
              test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
              interval: 30s
  
      flower:
          image: puckel/docker-airflow:1.10.2
          restart: always
          depends_on:
              - redis
          environment:
              - EXECUTOR=Celery
          ports:
              - "5555:5555"
          command: flower
  
      scheduler:
          image: puckel/docker-airflow:1.10.2
          restart: always
          depends_on:
              - webserver
          volumes:
              - ./dags:/usr/local/airflow/dags
              - /data/qa/druid-data/airflow/dags:/usr/local/airflow/dags
          environment:
              - LOAD_EX=n
              - FERNET_KEY=&lt;&gt;
              - EXECUTOR=Celery
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
          command: scheduler
  
      worker:
          image: puckel/docker-airflow:1.10.2
          restart: always
          depends_on:
              - scheduler
          volumes:
              - ./dags:/usr/local/airflow/dags
              - /data/qa/druid-data/airflow/dags:/usr/local/airflow/dags
          environment:
              - FERNET_KEY=&lt;&gt;
              - EXECUTOR=Celery
              - POSTGRES_USER=airflow
              - POSTGRES_PASSWORD=airflow
              - POSTGRES_DB=airflow
          command: worker -q druid_queue
  </code></pre>
  
  <p>The variable in server 1 looks like<br>
  broker_url=redis://redis:6379/1
  result_backend=db+postgresql://airflow:airflow@postgres:5432/airflow</p>
  
  <p>The variable in server 2 looks like<br>
  broker_url=redis://10.0.11.4:6999/1
  result_backend=db+postgresql://airflow:airflow@10.0.11.4:5434/airflow</p>
  
  <p>Is there something wrong with my configurations.
  when I run the dag from the webserver of server A the dag is stuck:<br>
  <a href="https://i.stack.imgur.com/iKN8c.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iKN8c.png" alt="enter image description here"></a> </p>
  
  <p>The logs captured in the scheduler of Server A container:   </p>
  
  <pre><code>[2019-04-12 14:42:35,184] {{jobs.py:1215}} INFO - Setting the follow tasks to queued state:
      &lt;TaskInstance: dummy_for_testing.print_date 2019-04-12 14:42:33.552786+00:00 [scheduled]&gt;
  [2019-04-12 14:42:35,194] {{jobs.py:1299}} INFO - Setting the following 1 tasks to queued state:
      &lt;TaskInstance: dummy_for_testing.print_date 2019-04-12 14:42:33.552786+00:00 [queued]&gt;
  [2019-04-12 14:42:35,194] {{jobs.py:1341}} INFO - Sending ('dummy_for_testing', 'print_date', datetime.datetime(2019, 4, 12, 14, 42, 33, 552786, tzinfo=&lt;TimezoneInfo [UTC, GMT, +00:00:00, STD]&gt;), 1) to executor with priority 3 and queue default
  [2019-04-12 14:42:35,194] {{base_executor.py:56}} INFO - Adding to queue: airflow run dummy_for_testing print_date 2019-04-12T14:42:33.552786+00:00 --local -sd /usr/local/airflow/dags/dag_test.py
  [2019-04-12 14:42:35,199] {{celery_executor.py:83}} INFO - [celery] queuing ('dummy_for_testing', 'print_date', datetime.datetime(2019, 4, 12, 14, 42, 33, 552786, tzinfo=&lt;TimezoneInfo [UTC, GMT, +00:00:00, STD]&gt;), 1) through celery, queue=default
  [2019-04-12 14:42:37,152] {{jobs.py:1559}} INFO - Harvesting DAG parsing results
  [2019-04-12 14:42:39,154] {{jobs.py:1559}} INFO - Harvesting DAG parsing results
  [2019-04-12 14:42:40,610] {{sqlalchemy.py:79}} WARNING - DB connection invalidated. Reconnecting...
  [2019-04-12 14:42:41,156] {{jobs.py:1559}} INFO - Harvesting DAG parsing results
  [2019-04-12 14:42:41,179] {{jobs.py:1106}} INFO - 1 tasks up for execution:
      &lt;TaskInstance: dummy_for_testing.print_host 2019-04-12 14:42:33.552786+00:00 [scheduled]&gt;
  [2019-04-12 14:42:41,182] {{jobs.py:1141}} INFO - Figuring out tasks to run in Pool(name=None) with 128 open slots and 1 task instances in queue
  [2019-04-12 14:42:41,184] {{jobs.py:1177}} INFO - DAG dummy_for_testing has 12/16 running and queued tasks
  [2019-04-12 14:42:41,184] {{jobs.py:1215}} INFO - Setting the follow tasks to queued state:
      &lt;TaskInstance: dummy_for_testing.print_host 2019-04-12 14:42:33.552786+00:00 [scheduled]&gt;
  [2019-04-12 14:42:41,193] {{jobs.py:1299}} INFO - Setting the following 1 tasks to queued state:
      &lt;TaskInstance: dummy_for_testing.print_host 2019-04-12 14:42:33.552786+00:00 [queued]&gt;
  [2019-04-12 14:42:41,193] {{jobs.py:1341}} INFO - Sending ('dummy_for_testing', 'print_host', datetime.datetime(2019, 4, 12, 14, 42, 33, 552786, tzinfo=&lt;TimezoneInfo [UTC, GMT, +00:00:00, STD]&gt;), 1) to executor with priority 2 and queue druid_queue
  [2019-04-12 14:42:41,194] {{base_executor.py:56}} INFO - Adding to queue: airflow run dummy_for_testing print_host 2019-04-12T14:42:33.552786+00:00 --local -sd /usr/local/airflow/dags/dag_test.py
  [2019-04-12 14:42:41,198] {{celery_executor.py:83}} INFO - [celery] queuing ('dummy_for_testing', 'print_host', datetime.datetime(2019, 4, 12, 14, 42, 33, 552786, tzinfo=&lt;TimezoneInfo [UTC, GMT, +00:00:00, STD]&gt;), 1) through celery, queue=druid_queue
  </code></pre>
  
  <p>Server A config:<br>
  <a href="https://i.stack.imgur.com/5IzKM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5IzKM.png" alt="enter image description here"></a><br>
  Server B config:
  <a href="https://i.stack.imgur.com/hF4ho.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hF4ho.png" alt="enter image description here"></a><br>
  Sserver A celery Broker<br>
  <a href="https://i.stack.imgur.com/8Z54M.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8Z54M.png" alt="enter image description here"></a></p>