<p>I have airflow instance deployed on kubernetes cluster using bitnami helm chart with Kubernetes Executor. If I try to execute a task which takes more than 5 mins, Scheduler automatically kills that task.</p>
<p>I have already tried [this][1] but does not work</p>
<p>Scheduler Logs:</p>
<pre><code>[2022-08-05 18:27:20,117] {kubernetes_executor.py:147} INFO - Event: process1800.404b6cc740064e90955e28dd736f899e had an event of type MODIFIED
[2022-08-05 18:27:20,118] {kubernetes_executor.py:213} INFO - Event: process1800.404b6cc740064e90955e28dd736f899e is Running
[2022-08-05 18:27:38,908] {kubernetes_executor.py:454} INFO - Found 0 queued task instances
[2022-08-05 18:28:39,212] {kubernetes_executor.py:454} INFO - Found 0 queued task instances
[2022-08-05 18:29:39,394] {kubernetes_executor.py:454} INFO - Found 0 queued task instances
[2022-08-05 18:30:39,575] {kubernetes_executor.py:454} INFO - Found 0 queued task instances
[2022-08-05 18:31:39,766] {kubernetes_executor.py:454} INFO - Found 0 queued task instances
[2022-08-05 18:32:10,603] {kubernetes_executor.py:147} INFO - Event: process1800.404b6cc740064e90955e28dd736f899e had an event of type MODIFIED
[2022-08-05 18:32:10,603] {kubernetes_executor.py:213} INFO - Event: process1800.404b6cc740064e90955e28dd736f899e is Running
[2022-08-05 18:32:11,436] {kubernetes_executor.py:147} INFO - Event: process1800.404b6cc740064e90955e28dd736f899e had an event of type MODIFIED
[2022-08-05 18:32:11,436] {kubernetes_executor.py:206} INFO - Event: process1800.404b6cc740064e90955e28dd736f899e Succeeded
[2022-08-05 18:32:11,943] {kubernetes_executor.py:374} INFO - Attempting to finish pod; pod_id: process1800.404b6cc740064e90955e28dd736f899e; state: None; annotations: {'dag_id': 'kubernetes_pod', 'task_id': 'process1800', 'execution_date': None, 'run_id': 'manual__2022-08-05T18:26:58.893963+00:00', 'try_number': '1'}```

Worker Log

/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py:357 DeprecationWarning: The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
[2022-08-05 18:27:20,359] {dagbag.py:500} INFO - Filling up the DagBag from /opt/bitnami/airflow/dags/git_dataextractor-dag/docker_scraplimit.py
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/kubernetes/volume.py:22 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/kubernetes/volume_mount.py:22 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
Running &lt;TaskInstance: kubernetes_pod.process1800 manual__2022-08-05T18:26:58.893963+00:00 [queued]&gt; on host process1800.404b6cc740064e90955

Process finished with exit code 0


Task Log:

[2022-08-05, 18:32:09 UTC] {local_task_job.py:82} ERROR - Received SIGTERM. Terminating subprocesses
[2022-08-05, 18:32:09 UTC] {process_utils.py:120} INFO - Sending Signals.SIGTERM to group 18. PIDs of all processes in the group: [18]
[2022-08-05, 18:32:09 UTC] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 18
[2022-08-05, 18:32:09 UTC] {taskinstance.py:1430} ERROR - Received SIGTERM. Terminating subprocesses.
[2022-08-05, 18:32:09 UTC] {kubernetes_pod.py:459} INFO - Deleting pod: airflow-yield-datalayer.7859414c89794502b683608a386a819c
[2022-08-05, 18:32:09 UTC] {taskinstance.py:1774} ERROR - Task failed with exception
Traceback (most recent call last):
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py&quot;, line 417, in execute
    self.pod_manager.follow_container_logs(
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py&quot;, line 208, in follow_container_logs
    last_log_time = follow_logs(since_time=last_log_time)
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py&quot;, line 195, in follow_logs
    for line in logs:  # type: bytes
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/response.py&quot;, line 811, in __iter__
    for chunk in self.stream(decode_content=True):
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/response.py&quot;, line 575, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/response.py&quot;, line 767, in read_chunked
    self._update_chunk_length()
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/response.py&quot;, line 697, in _update_chunk_length
    line = self._fp.fp.readline()
  File &quot;/opt/bitnami/python/lib/python3.8/socket.py&quot;, line 669, in readinto
    return self._sock.recv_into(b)
  File &quot;/opt/bitnami/python/lib/python3.8/ssl.py&quot;, line 1241, in recv_into
    return self.read(nbytes, buffer)
  File &quot;/opt/bitnami/python/lib/python3.8/ssl.py&quot;, line 1099, in read
    return self._sslobj.read(len, buffer)
  File &quot;/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py&quot;, line 1432, in signal_handler
    raise AirflowException(&quot;Task received SIGTERM signal&quot;)
airflow.exceptions.AirflowException: Task received SIGTERM signal


  [1]: https://github.com/apache/airflow/issues/18041
</code></pre>

Abhijeet Khandelwal
<p>In bitnami helm chart, there is an option to add a <code>livenessProb</code> to the pod_template (the template used to create a new pod/worker).</p>
<p>This <code>livenessProbe</code> is <a href="https://github.com/bitnami/charts/blob/master/bitnami/airflow/values.yaml#L694-L700" rel="nofollow noreferrer">enabled</a> by default, with <code>initialDelaySeconds=180s</code> (3 min), 20 seconds between each two tries, and 6 as failure threshold, so 2 min for the checking and 3 min before the first try, which is 5 min in total.</p>
<p>You can check what is the problem with this Prob and solving it, or just disable the Prob by setting:</p>
<pre class="lang-yaml prettyprint-override"><code>worker:
  livenessProbe:
    enabled: false
</code></pre>
