title: Apache Airflow: Executor reports task instance finished (failed) although the task says its queued
body: <p>Our airflow installation is using CeleryExecutor.
  The concurrency configs were </p>
  
  <pre><code># The amount of parallelism as a setting to the executor. This defines
  # the max number of task instances that should run simultaneously
  # on this airflow installation
  parallelism = 16
  
  # The number of task instances allowed to run concurrently by the scheduler
  dag_concurrency = 16
  
  # Are DAGs paused by default at creation
  dags_are_paused_at_creation = True
  
  # When not using pools, tasks are run in the "default pool",
  # whose size is guided by this config element
  non_pooled_task_slot_count = 64
  
  # The maximum number of active DAG runs per DAG
  max_active_runs_per_dag = 16
  [celery]
  # This section only applies if you are using the CeleryExecutor in
  # [core] section above
  
  # The app name that will be used by celery
  celery_app_name = airflow.executors.celery_executor
  
  # The concurrency that will be used when starting workers with the
  # "airflow worker" command. This defines the number of task instances that
  # a worker will take, so size up your workers based on the resources on
  # your worker box and the nature of your tasks
  celeryd_concurrency = 16
  
  </code></pre>
  
  <p>We have a dag that executes daily. It has around some tasks in parallel following a pattern that senses whether the data exists in hdfs then sleep 10 mins, and finally upload to s3.</p>
  
  <p>Some of the tasks has been encountering the following error:</p>
  
  <pre><code>2019-05-12 00:00:46,212 ERROR - Executor reports task instance &lt;TaskInstance: example_dag.task1 2019-05-11 04:00:00+00:00 [queued]&gt; finished (failed) although the task says its queued. Was the task killed externally?
  2019-05-12 00:00:46,558 INFO - Marking task as UP_FOR_RETRY
  2019-05-12 00:00:46,561 WARNING - section/key [smtp/smtp_user] not found in config
  </code></pre>
  
  <p>This kind of error occurs randomly in those tasks. When this error happens, the state of task instance is immediately set to up_for_retry, and no logs in the worker nodes. After some retries, they execute and finished eventually.</p>
  
  <p>This problem sometimes gives us large ETL delay. Anyone knows how to solve this problem?</p>