title: How to safely restart Airflow and kill a long-running task?
body: <p>I have Airflow is running in Kubernetes using the CeleryExecutor. Airflow submits and monitors Spark jobs using the <a href="https://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/databricks_operator.html" rel="nofollow noreferrer">DatabricksOperator</a>.</p>
  <p>My streaming Spark jobs have a very long runtime (they run forever unless they fail or are cancelled). When pods for Airflow worker are killed while a streaming job is running, the following happens:</p>
  <ol>
  <li>Associated task becomes a zombie (running state, but no process with heartbeat)</li>
  <li>Task is marked as failed when Airflow reaps zombies</li>
  <li>Spark streaming job continues to run</li>
  </ol>
  <p><strong>How can I force the worker to kill my Spark job before it shuts down?</strong></p>
  <p>I've tried killing the Celery worker with a TERM signal, but apparently that causes Celery to stop accepting new tasks and wait for current tasks to finish (<a href="https://docs.celeryproject.org/en/stable/userguide/workers.html#process-signals" rel="nofollow noreferrer">docs</a>).</p>