<p>My airflow service runs as a kubernetes deployment, and has two containers, one for the <code>webserver</code> and one for the <code>scheduler</code>.
I'm running a task using a KubernetesPodOperator, with <code>in_cluster=True</code> parameters, and it runs well, I can even <code>kubectl logs pod-name</code> and all the logs show up. </p>

<p>However, the <code>airflow-webserver</code> is unable to fetch the logs:</p>

<pre><code>*** Log file does not exist: /tmp/logs/dag_name/task_name/2020-05-19T23:17:33.455051+00:00/1.log
*** Fetching from: http://pod-name-7dffbdf877-6mhrn:8793/log/dag_name/task_name/2020-05-19T23:17:33.455051+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='pod-name-7dffbdf877-6mhrn', port=8793): Max retries exceeded with url: /log/dag_name/task_name/2020-05-19T23:17:33.455051+00:00/1.log (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7fef6e00df10&gt;: Failed to establish a new connection: [Errno 111] Connection refused'))
</code></pre>

<p>It seems as the pod is unable to connect to the airflow logging service, on port 8793. If I <code>kubectl exec bash</code> into the container, I can curl localhost on port 8080, but not on 80 and 8793.</p>

<p>Kubernetes deployment:</p>

<pre><code># Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-name
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-name
  template:
    metadata:
      labels:
        app: pod-name
    spec:
      restartPolicy: Always
      volumes:
        - name: airflow-cfg
          configMap:
            name: airflow.cfg
        - name: dags
          emptyDir: {}
      containers:
      - name: airflow-scheduler
        args:
        - airflow
        - scheduler
        image: registry.personal.io:5000/image/path
        imagePullPolicy: Always
        volumeMounts:
        - name: dags
          mountPath: /airflow_dags
        - name: airflow-cfg
          mountPath: /home/airflow/airflow.cfg
          subPath: airflow.cfg
        env:
        - name: EXECUTOR
          value: Local
        - name: LOAD_EX
          value: "n"
        - name: FORWARDED_ALLOW_IPS
          value: "*"
        ports:
          - containerPort: 8793
          - containerPort: 8080
      - name: airflow-webserver
        args:
        - airflow
        - webserver
        - --pid
        - /tmp/airflow-webserver.pid
        image: registry.personal.io:5000/image/path
        imagePullPolicy: Always
        volumeMounts:
        - name: dags
          mountPath: /airflow_dags
        - name: airflow-cfg
          mountPath: /home/airflow/airflow.cfg
          subPath: airflow.cfg
        ports:
        - containerPort: 8793
        - containerPort: 8080
        env:
        - name: EXECUTOR
          value: Local
        - name: LOAD_EX
          value: "n"
        - name: FORWARDED_ALLOW_IPS
          value: "*"
</code></pre>

<p>note: If airflow is run in dev environment (locally instead of kubernetes) it all works perfectly.</p>

Yuzobra
<p>Creating a Persistent Volume and storing logs on them might help.</p>
<pre><code>--
kind: PersistentVolume
apiVersion: v1
metadata:
  name: testlog-volume
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 2Gi
  hostPath:
    path: /opt/airflow/logs/
  storageClassName: standard
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: testlog-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: standard
</code></pre>
<p>if you are using helm chart to deploy airflow, you can use</p>
<pre><code> --set executor=KubernetesExecutor --set logs.persistence.enabled=true --set logs.persistence.existingClaim=testlog-volume
</code></pre>
