<p>We are facing strange issue with EKS Fargate Pods. We want to push logs to cloudwatch with sidecar fluent-bit container and for that we are mounting the separately created <code>/logs/boot</code> and <code>/logs/access</code> folders on both the containers with <code>emptyDir: {}</code> type. But somehow the <code>access</code> folder is getting deleted. When we tested this setup in local docker it produced desired results and things were working fine but not when deployed in the EKS fargate. Below is our manifest files</p>
<p><strong>Dockerfile</strong></p>
<pre><code>FROM anapsix/alpine-java:8u201b09_server-jre_nashorn

ARG LOG_DIR=/logs

# Install base packages
RUN apk update
RUN apk upgrade
# RUN apk add ca-certificates &amp;&amp; update-ca-certificates

# Dynamically set the JAVA_HOME path
RUN export JAVA_HOME=&quot;$(dirname $(dirname $(readlink -f $(which java))))&quot; &amp;&amp; echo $JAVA_HOME

# Add Curl
RUN apk --no-cache add curl

RUN mkdir -p $LOG_DIR/boot $LOG_DIR/access
RUN chmod -R 0777 $LOG_DIR/*

# Add metadata to the image to describe which port the container is listening on at runtime.

# Change TimeZone
RUN apk add --update tzdata
ENV TZ=&quot;Asia/Kolkata&quot;

# Clean APK cache
RUN rm -rf /var/cache/apk/*

# Setting JAVA HOME
ENV JAVA_HOME=/opt/jdk

# Copy all files and folders
COPY . .
RUN rm -rf /opt/jdk/jre/lib/security/cacerts
COPY cacerts /opt/jdk/jre/lib/security/cacerts
COPY standalone.xml /jboss-eap-6.4-integration/standalone/configuration/

# Set the working directory.
WORKDIR /jboss-eap-6.4-integration/bin

EXPOSE 8177

CMD [&quot;./erctl&quot;]
</code></pre>
<p><strong>Deployment</strong></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: vinintegrator
  namespace: eretail
  labels:
    app: vinintegrator
    pod: fargate
spec:
  selector:
    matchLabels:
      app: vinintegrator
      pod: fargate
  replicas: 2
  template:
    metadata:
      labels:
        app: vinintegrator
        pod: fargate
    spec:
      securityContext:
        fsGroup: 0
      serviceAccount: eretail
      containers:
      - name: vinintegrator
        imagePullPolicy: IfNotPresent
        image: 653580443710.dkr.ecr.ap-southeast-1.amazonaws.com/vinintegrator-service:latest
        resources:
          limits:
            memory: &quot;7629Mi&quot;
            cpu: &quot;1.5&quot;
          requests:
            memory: &quot;5435Mi&quot;
            cpu: &quot;750m&quot;
        ports:
        - containerPort: 8177
          protocol: TCP
        # securityContext:
          # runAsUser: 506
          # runAsGroup: 506
        volumeMounts:
          - mountPath: /jboss-eap-6.4-integration/bin
            name: bin
          - mountPath: /logs
            name: logs
      - name: fluent-bit
        image: 657281243710.dkr.ecr.ap-southeast-1.amazonaws.com/fluent-bit:latest
        imagePullPolicy: IfNotPresent
        env:
          - name: HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 200m
            memory: 100Mi
        volumeMounts:
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: logs
          mountPath: /logs
          readOnly: true
      volumes:
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config
        - name: logs
          emptyDir: {}
        - name: bin
          persistentVolumeClaim:
            claimName: vinintegrator-pvc
</code></pre>
<p>Below is the /logs folder ownership and permission. <strong>Please notice the 's' in <code>drwxrwsrwx</code></strong></p>
<pre><code>drwxrwsrwx    3 root     root          4096 Oct  1 11:50 logs
</code></pre>
<p>Below is the content inside logs folder. <strong>Please notice the access folder is not created or deleted.</strong></p>
<pre><code>/logs # ls -lrt
total 4
drwxr-sr-x    2 root     root          4096 Oct  1 11:50 boot
/logs #
</code></pre>
<p>Below is the configmap of Fluent-Bit</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: eretail
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020
        
    @INCLUDE application-log.conf
  
  application-log.conf: |
    [INPUT]
        Name                tail
        Path                /logs/boot/*.log
        Tag                 boot
        
    [INPUT]
        Name                tail
        Path                /logs/access/*.log
        Tag                 access
        
    [OUTPUT]
        Name                cloudwatch_logs
        Match               *boot*
        region              ap-southeast-1
        log_group_name      eks-fluent-bit
        log_stream_prefix   boot-log-
        auto_create_group   On
        
    [OUTPUT]
        Name                cloudwatch_logs
        Match               *access*
        region              ap-southeast-1
        log_group_name      eks-fluent-bit
        log_stream_prefix   access-log-
        auto_create_group   On
        
  parsers.conf: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ
</code></pre>
<p>Below is error log of Fluent-bit container</p>
<pre><code>AWS for Fluent Bit Container Image Version 2.14.0
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/01 06:20:33] [ info] [engine] started (pid=1)
[2021/10/01 06:20:33] [ info] [storage] version=1.1.1, initializing...
[2021/10/01 06:20:33] [ info] [storage] in-memory
[2021/10/01 06:20:33] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/01 06:20:33] [error] [input:tail:tail.1] read error, check permissions: /logs/access/*.log
[2021/10/01 06:20:33] [ warn] [input:tail:tail.1] error scanning path: /logs/access/*.log
[2021/10/01 06:20:38] [error] [net] connection #33 timeout after 5 seconds to: 169.254.169.254:80
[2021/10/01 06:20:38] [error] [net] socket #33 could not connect to 169.254.169.254:80
</code></pre>

Nitin G
<p>Suggest remove the following from your Dockerfile:</p>
<pre><code>RUN mkdir -p $LOG_DIR/boot $LOG_DIR/access
RUN chmod -R 0777 $LOG_DIR/*
</code></pre>
<p>Use the following method to setup the log directories and permissions:</p>
<pre><code>apiVersion: v1  
kind: Pod    # Deployment
metadata:
  name: busy
  labels:
    app: busy
spec:
  volumes:
  - name: logs  # Shared folder with ephemeral storage
    emptyDir: {}

  initContainers:    # Setup your log directory here
  - name: setup
    image: busybox
    command: [&quot;bin/ash&quot;, &quot;-c&quot;]
    args:
    - &gt;
      mkdir -p /logs/boot /logs/access;
      chmod -R 777 /logs
    volumeMounts:
    - name: logs
      mountPath: /logs

  containers:
  - name: app    # Run your application and logs to the directories
    image: busybox
    command: [&quot;bin/ash&quot;,&quot;-c&quot;]
    args:
    - &gt;
      while :; do echo &quot;$(date): $(uname -r)&quot; | tee -a /logs/boot/boot.log /logs/access/access.log; sleep 1; done
    volumeMounts:
    - name: logs
      mountPath: /logs

  - name: logger    # Any logger that you like
    image: busybox
    command: [&quot;bin/ash&quot;,&quot;-c&quot;]
    args:           # tail the app logs, forward to CW etc...
    - &gt;
      sleep 5;
      tail -f /logs/boot/boot.log /logs/access/access.log
    volumeMounts:
    - name: logs
      mountPath: /logs
</code></pre>
<p>The snippet runs on Fargate as well, run <code>kubectl logs -f busy -c logger</code> to see the tailing. In real world, the &quot;app&quot; is your java app, &quot;logger&quot; is any log agent you desired. Note Fargate has <a href="https://aws.amazon.com/blogs/containers/fluent-bit-for-amazon-eks-on-aws-fargate-is-here/" rel="nofollow noreferrer">native logging capability</a> using AWS Fluent-bit, you do not need to run AWS Fluent-bit as sidecar.</p>
