title: Airflow installation with celery - Task fails without executing it - raise AirflowException('Celery command failed')
body: <p>I have installed Airflow with celeryexecutor having 2 workers , 1 scheduler and 1 webserver. I am starting all of these components with systemd. All components start fine I can see the DAGs in web UI. When I execute any DAG first task itself fails without even starting it. Seems some issue in starting the task at worker. Below is the log I am getting in journalctl.</p>
  
  <p>Worker error log : </p>
  
  <pre><code>ct 21 19:17:23 mydomain.mycompany python[9780]: [2019-10-21 19:17:23,425: DEBUG/MainProcess] Task accepted: airflow.executors.celery_executor.execute_command[526ed99b-f21f-40ba-8018-afbf0dc9a36a] pid:9800
  Oct 21 19:17:23 mydomain.mycompany python[9780]: [2019-10-21 19:17:23,425: INFO/Worker-16] Executing command in Celery: airflow run sip_data_parsing_batch_execution_dag clear_configurations 2019-10-21T19:17:20.802983+00:00 --local -sd /opt/sip-orchestration/airflow_pai/dags/sip_data_parsing/sip_data_parsing_dag.py
  Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2618, in _execute
  Oct 21 19:17:24 mydomain.mycompany python[9780]: self.task_runner.start()
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/bash_task_runner.py", line 34, in start
  Oct 21 19:17:24 mydomain.mycompany python[9780]: self.process = self.run_command(['bash', '-c'], join_args=True)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 126, in run_command
  Oct 21 19:17:24 mydomain.mycompany python[9780]: preexec_fn=os.setsid
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
  Oct 21 19:17:24 mydomain.mycompany python[9780]: restore_signals, start_new_session)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
  Oct 21 19:17:24 mydomain.mycompany python[9780]: raise child_exception_type(errno_num, err_msg, err_filename)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: FileNotFoundError: [Errno 2] No such file or directory: 'bash': 'bash'
  Oct 21 19:17:24 mydomain.mycompany python[9780]: During handling of the above exception, another exception occurred:
  Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/bin/airflow", line 32, in &lt;module&gt;
  Oct 21 19:17:24 mydomain.mycompany python[9780]: args.func(args)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
  Oct 21 19:17:24 mydomain.mycompany python[9780]: return f(*args, **kwargs)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/bin/cli.py", line 490, in run
  Oct 21 19:17:24 mydomain.mycompany python[9780]: _run(args, dag, ti)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/bin/cli.py", line 401, in _run
  Oct 21 19:17:24 mydomain.mycompany python[9780]: run_job.run()
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 203, in run
  Oct 21 19:17:24 mydomain.mycompany python[9780]: self._execute()
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2654, in _execute
  Oct 21 19:17:24 mydomain.mycompany python[9780]: self.on_kill()
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2658, in on_kill
  Oct 21 19:17:24 mydomain.mycompany python[9780]: self.task_runner.on_finish()
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/bash_task_runner.py", line 44, in on_finish
  Oct 21 19:17:24 mydomain.mycompany python[9780]: super(BashTaskRunner, self).on_finish()
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 163, in on_finish
  Oct 21 19:17:24 mydomain.mycompany python[9780]: subprocess.call(['sudo', 'rm', self._cfg_path], close_fds=True)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 287, in call
  Oct 21 19:17:24 mydomain.mycompany python[9780]: with Popen(*popenargs, **kwargs) as p:
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
  Oct 21 19:17:24 mydomain.mycompany python[9780]: restore_signals, start_new_session)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
  Oct 21 19:17:24 mydomain.mycompany python[9780]: raise child_exception_type(errno_num, err_msg, err_filename)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: FileNotFoundError: [Errno 2] No such file or directory: 'sudo': 'sudo'
  Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,681: ERROR/Worker-16] execute_command encountered a CalledProcessError
  Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 58, in execute_command
  Oct 21 19:17:24 mydomain.mycompany python[9780]: close_fds=True, env=env)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
  Oct 21 19:17:24 mydomain.mycompany python[9780]: raise CalledProcessError(retcode, cmd)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: subprocess.CalledProcessError: Command 'airflow run sip_data_parsing_batch_execution_dag clear_configurations 2019-10-21T19:17:20.802983+00:00 --local -sd /opt/sip-orchestration/airflow_pai/dags/sip_data_parsing/sip_data_parsing_dag.py' returned non-zero exit status 1.
  Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,682: ERROR/Worker-16] None
  Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,684: ERROR/MainProcess] Task airflow.executors.celery_executor.execute_command[526ed99b-f21f-40ba-8018-afbf0dc9a36a] raised unexpected: AirflowException('Celery command failed',)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/celery/app/trace.py", line 240, in trace_task
  Oct 21 19:17:24 mydomain.mycompany python[9780]: R = retval = fun(*args, **kwargs)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
  Oct 21 19:17:24 mydomain.mycompany python[9780]: return self.run(*args, **kwargs)
  Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 63, in execute_command
  Oct 21 19:17:24 mydomain.mycompany python[9780]: raise AirflowException('Celery command failed')
  Oct 21 19:17:24 mydomain.mycompany python[9780]: airflow.exceptions.AirflowException: Celery command failed
  
  
  </code></pre>
  
  <p>Command I use to start components.</p>
  
  <ul>
  <li>sudo service airflow-worker start</li>
  <li><p>sudo service airflow-scheduler start</p></li>
  <li><p>sudo service airflow-webserver start</p></li>
  </ul>
  
  <p>Airflow version - 1.10.1
  Celery version - 3.1.17</p>