title: Airflow DAG randomly fails with timeout
body: <p>I have a DAG on Airflow (version: 1.10.15) which fails seemingly randomly, even though the code is not changed in between the different runs. It is a training script which normally should not take more than 2 hours, but occasionally it runs without ever completing, and eventually times out at the maximum set timeout.</p>
  <p>When checking the logs, I'm seeing a continuous iteration of the following:</p>
  <pre><code>[2022-05-31 12:51:13,219] {databricks_operator.py:89} INFO - &lt;SCRIPT_NAME&gt; in run state: {'life_cycle_state': 
  'RUNNING', 'result_state': None, 'state_message': 'In run'}
  [2022-05-31 12:51:13,219] {databricks_operator.py:90} INFO - View run status, Spark UI, and logs at &lt;LOG_URL&gt;
  [2022-05-31 12:51:13,219] {databricks_operator.py:91} INFO - Sleeping for 30 seconds.
  </code></pre>
  <p>So there is seemingly no indication of failure in the logs. I then checked the actual notebook file, and saw that the execution of DAG stalls at the training step; one or two epochs are completed but then, no more training seems to occur until the timeout.</p>
  <p>I'm pretty new to using Airflow, so at this point I'm not even sure where I should begin debugging this issue. If this was some error in the training script, it should give an error consistently, I presume. But in most cases the DAG actually runs correctly, it just stalls in an endless loop and times out only occasionally.</p>
  <p>What might be going wrong?</p>