<p>I am running Airflow on Kubernetes from the <a href="https://github.com/helm/charts/tree/master/stable/airflow" rel="nofollow noreferrer">stable helm chart</a>. I'm running this in an AWS environment. This error exists with and without mounting any external volumes for log storage. I tried to set the configuration of the [logs] section to point to an EFS volume that I created. The PV gets mounted through a PVC but my containers are crashing (scheduler and web) due to the following error:</p>
<pre><code>*** executing Airflow initdb...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.6/logging/config.py&quot;, line 565, in configure
    handler = self.configure_handler(handlers[name])
  File &quot;/usr/local/lib/python3.6/logging/config.py&quot;, line 738, in configure_handler
    result = factory(**kwargs)
  File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py&quot;, line 50, in __init__
    os.makedirs(self._get_log_directory())
  File &quot;/usr/local/lib/python3.6/os.py&quot;, line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/home/airflow/.local/bin/airflow&quot;, line 25, in &lt;module&gt;
    from airflow.configuration import conf
  File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py&quot;, line 47, in &lt;module&gt;
    settings.initialize()
  File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py&quot;, line 374, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py&quot;, line 68, in configure_logging
    raise e
  File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py&quot;, line 63, in configure_logging
    dictConfig(logging_config)
  File &quot;/usr/local/lib/python3.6/logging/config.py&quot;, line 802, in dictConfig
    dictConfigClass(config).configure()
  File &quot;/usr/local/lib/python3.6/logging/config.py&quot;, line 573, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
</code></pre>
<p>Persistent volume (created manually not from the stable/airflow chart)</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;kind&quot;:&quot;PersistentVolume&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;efs-pv&quot;},&quot;spec&quot;:{&quot;accessModes&quot;:[&quot;ReadWriteMany&quot;],&quot;capacity&quot;:{&quot;storage&quot;:&quot;5Gi&quot;},&quot;csi&quot;:{&quot;driver&quot;:&quot;efs.csi.aws.com&quot;,&quot;volumeHandle&quot;:&quot;fs-e476a166&quot;},&quot;persistentVolumeReclaimPolicy&quot;:&quot;Retain&quot;,&quot;storageClassName&quot;:&quot;efs-sc&quot;,&quot;volumeMode&quot;:&quot;Filesystem&quot;}}
    pv.kubernetes.io/bound-by-controller: &quot;yes&quot;
  creationTimestamp: &quot;2020-08-20T15:47:21Z&quot;
  finalizers:
  - kubernetes.io/pv-protection
  name: efs-pv
  resourceVersion: &quot;49476860&quot;
  selfLink: /api/v1/persistentvolumes/efs-pv
  uid: 45d9f5ea-66c1-493e-a2f5-03e17f397747
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: efs-claim
    namespace: airflow
    resourceVersion: &quot;49476857&quot;
    uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-e476a166
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  volumeMode: Filesystem
status:
  phase: Bound
</code></pre>
<p>Persistent Volume Claim for logs (created manually not from the stable/airflow chart):</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;kind&quot;:&quot;PersistentVolumeClaim&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;efs-claim&quot;,&quot;namespace&quot;:&quot;airflow&quot;},&quot;spec&quot;:{&quot;accessModes&quot;:[&quot;ReadWriteMany&quot;],&quot;resources&quot;:{&quot;requests&quot;:{&quot;storage&quot;:&quot;5Gi&quot;}},&quot;storageClassName&quot;:&quot;efs-sc&quot;}}
    pv.kubernetes.io/bind-completed: &quot;yes&quot;
    pv.kubernetes.io/bound-by-controller: &quot;yes&quot;
  creationTimestamp: &quot;2020-08-20T15:47:46Z&quot;
  finalizers:
  - kubernetes.io/pvc-protection
  name: efs-claim
  namespace: airflow
  resourceVersion: &quot;49476866&quot;
  selfLink: /api/v1/namespaces/airflow/persistentvolumeclaims/efs-claim
  uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: efs-sc
  volumeMode: Filesystem
  volumeName: efs-pv
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  phase: Bound
</code></pre>
<p>My <code>values.yaml</code> below:</p>
<pre><code>airflow:
  image:
    repository: apache/airflow
    tag: 1.10.10-python3.6
    ## values: Always or IfNotPresent
    pullPolicy: IfNotPresent
    pullSecret: &quot;&quot;

  executor: KubernetesExecutor

  fernetKey: &quot;XXXXXXXXXHIVb8jK6lfmSAvx4mO6Arehnc=&quot;

  config:
    AIRFLOW__CORE__REMOTE_LOGGING: &quot;True&quot;
    AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: &quot;s3://mybucket/airflow/logs&quot;
    AIRFLOW__CORE__REMOTE_LOG_CONN_ID: &quot;MyS3Conn&quot;
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: &quot;apache/airflow&quot;
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: &quot;1.10.10-python3.6&quot;
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: &quot;IfNotPresent&quot;
    AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: &quot;10&quot;
    AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: &quot;efs-claim&quot;
    AIRFLOW__KUBERNETES__GIT_REPO: &quot;git@github.com:org/myrepo.git&quot;
    AIRFLOW__KUBERNETES__GIT_BRANCH: &quot;develop&quot;
    AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: &quot;/opt/airflow/dags&quot;
    AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: &quot;repo/&quot;
    AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: &quot;airflow-git-keys&quot;
    AIRFLOW__KUBERNETES__NAMESPACE: &quot;airflow&quot;
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: &quot;True&quot;
    AIRFLOW__KUBERNETES__RUN_AS_USER: &quot;50000&quot;
    AIRFLOW__CORE__LOAD_EXAMPLES: &quot;False&quot;
    AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: &quot;60&quot;
    AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: &quot;airflow&quot;

  podAnnotations: {}
  extraEnv: []
  extraConfigmapMounts: []
  extraContainers: []
  extraPipPackages: []
  extraVolumeMounts: []
  extraVolumes: []
scheduler:
  resources: {}
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  podDisruptionBudget:
    enabled: true
    maxUnavailable: &quot;100%&quot;
    minAvailable: &quot;&quot;
  connections:
    - id: MyS3Conn
      type: aws
      extra: |
        {
        &quot;aws_access_key_id&quot;: &quot;XXXXXXXXX&quot;,
        &quot;aws_secret_access_key&quot;: &quot;XXXXXXXX&quot;,
        &quot;region_name&quot;:&quot;us-west-1&quot;
        }

  refreshConnections: true
  variables: |
    {}

  pools: |
    {}

  numRuns: -1
  initdb: true
  preinitdb: false
  initialStartupDelay: 0
  extraInitContainers: []
web:
  resources: {}
  replicas: 1
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  service:
    annotations: {}
    sessionAffinity: &quot;None&quot;
    sessionAffinityConfig: {}
    type: ClusterIP
    externalPort: 8080
    loadBalancerIP: &quot;&quot;
    loadBalancerSourceRanges: []
    nodePort:
      http: &quot;&quot;

  baseUrl: &quot;http://localhost:8080&quot;
  serializeDAGs: false
  extraPipPackages: []
  initialStartupDelay: 0
  minReadySeconds: 5
  readinessProbe:
    enabled: false
    scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 3

  livenessProbe:
    enabled: true
    scheme: HTTP
    initialDelaySeconds: 300
    periodSeconds: 30
    timeoutSeconds: 3
    successThreshold: 1
    failureThreshold: 2

  secretsDir: /var/airflow/secrets
  secrets: []
  secretsMap:

workers:
  enabled: false
  resources: {}
  replicas: 1
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  autoscaling:
    enabled: false
    maxReplicas: 2
    metrics: []
  initialStartupDelay: 0
  celery:
    instances: 1
    gracefullTermination: false
    gracefullTerminationPeriod: 600
  terminationPeriod: 60
  secretsDir: /var/airflow/secrets
  secrets: []
  secretsMap:

flower:
  enabled: false
  resources: {}
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  basicAuthSecret: &quot;&quot;
  basicAuthSecretKey: &quot;&quot;
  urlPrefix: &quot;&quot;
  service:
    annotations: {}
    type: ClusterIP
    externalPort: 5555
    loadBalancerIP: &quot;&quot;
    loadBalancerSourceRanges: []
    nodePort:
      http: &quot;&quot;

  initialStartupDelay: 0
  extraConfigmapMounts: []

logs:
  path: /opt/airflow/logs
  persistence:
    enabled: true
    existingClaim: efs-claim
    subPath: &quot;&quot;
    storageClass: efs-sc
    accessMode: ReadWriteMany
    size: 1Gi
dags:
  path: /opt/airflow/dags
  doNotPickle: false
  installRequirements: false
  persistence:
    enabled: false
    existingClaim: &quot;&quot;
    subPath: &quot;&quot;
    storageClass: &quot;&quot;
    accessMode: ReadOnlyMany
    size: 1Gi
  git:
    url: git@github.com:org/myrepo.git
    ref: develop
    secret: airflow-git-keys
    sshKeyscan: false
    privateKeyName: id_rsa
    repoHost: github.com
    repoPort: 22
    gitSync:
      enabled: true
      resources: {}
      image:
        repository: alpine/git
        tag: latest
        pullPolicy: Always
      refreshTime: 60
  initContainer:
    enabled: false
    resources: {}
    image:
      repository: alpine/git
      tag: latest
      pullPolicy: Always
    mountPath: &quot;/dags&quot;
    syncSubPath: &quot;&quot;
ingress:
  enabled: false
  web:
    annotations: {}
    path: &quot;&quot;
    host: &quot;&quot;
    livenessPath: &quot;&quot;
    tls:
      enabled: false
      secretName: &quot;&quot;
    precedingPaths: []
    succeedingPaths: []
  flower:
    annotations: {}
    path: &quot;&quot;
    host: &quot;&quot;
    livenessPath: &quot;&quot;
    tls:
      enabled: false
      secretName: &quot;&quot;
rbac:
  create: true
serviceAccount:
  create: true
  name: &quot;&quot;
  annotations: {}
extraManifests: []

postgresql:

  enabled: true
  postgresqlDatabase: airflow
  postgresqlUsername: postgres
  postgresqlPassword: airflow
  existingSecret: &quot;&quot;
  existingSecretKey: &quot;postgresql-password&quot;
  persistence:
    enabled: true
    storageClass: &quot;&quot;
    accessModes:
      - ReadWriteOnce
    size: 5Gi

externalDatabase:
  type: postgres
  host: localhost
  port: 5432
  database: airflow
  user: airflow
  passwordSecret: &quot;&quot;
  passwordSecretKey: &quot;postgresql-password&quot;

redis:
  enabled: false
  password: airflow
  existingSecret: &quot;&quot;
  existingSecretKey: &quot;redis-password&quot;
  cluster:
    enabled: false
    slaveCount: 1
  master:
    resources: {}
    persistence:
      enabled: false
      storageClass: &quot;&quot;
      accessModes:
        - ReadWriteOnce

      size: 8Gi

  slave:
    resources: {}
    persistence:
      enabled: false
      storageClass: &quot;&quot;
      accessModes:
        - ReadWriteOnce

      size: 8Gi

externalRedis:
  host: localhost
  port: 6379
  databaseNumber: 1
  passwordSecret: &quot;&quot;
  passwordSecretKey: &quot;redis-password&quot;

serviceMonitor:
  enabled: false
  selector:
    prometheus: kube-prometheus
  path: /admin/metrics
  interval: &quot;30s&quot;

prometheusRule:
  enabled: false
  additionalLabels: {}
  groups: []
</code></pre>
<p>I'm not really sure what to do here if anyone knows how to fix the permission error.</p>

<p>I have had this issue with the Google Cloud Plateform and the helm airflow 1.2.0 chart (which uses airflow 2).
What ended up working was:</p>
<pre class="lang-yaml prettyprint-override"><code>extraInitContainers:
  - name: fix-volume-logs-permissions
    image: busybox
    command: [ &quot;sh&quot;, &quot;-c&quot;, &quot;chown -R 50000:0 /opt/airflow/logs/&quot; ]
    securityContext:
      runAsUser: 0
    volumeMounts:
      - mountPath: /opt/airflow/logs/
        name: logs
</code></pre>
<p>by tweaking based on Ajay's answer. Please note that:</p>
<ul>
<li>the values 50000:0 are based on uid and gid setup in your values.yaml</li>
<li>you need to use extraInitContainers under scheduler and not worker</li>
<li>&quot;logs&quot; seems to be the volume name automatically used by the helm logging config when enabled</li>
<li>Security context was necessary for me or else the chown failed due to unprivileged rights</li>
</ul>
