<p>The below container.conf works fine in Kubernetes 1.23 but fails after migrating to 1.25. I have also specified the deamonset that I have used to push the logs to cloudwatch. When I look into the logs of the fluentd deamonset I could see a lot of below errors</p>
<p>2023-04-03 01:32:06 +0000 [warn]: #0 [in_tail_container_logs] pattern not matched: &quot;2023-04-03T01:32:02.9256618Z stdout F [2023-04-03T01:32:02.925Z] DEBUG transaction-677fffdfc4-tc4rx-18/TRANSPORTER: NATS client pingTimer: 1&quot;</p>
<pre><code>

container.conf
==============
    &lt;source&gt;
      @type tail
      @id in_tail_container_logs
      @label @containers
      path /var/log/containers/*.log
      exclude_path [&quot;/var/log/containers/fluentd*&quot;]
      pos_file /var/log/fluentd-containers.log.pos
      tag *
      read_from_head true
      &lt;parse&gt;
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      &lt;/parse&gt;
    &lt;/source&gt;
    &lt;label @containers&gt;
      &lt;filter **&gt;
        @type kubernetes_metadata
        @id filter_kube_metadata
      &lt;/filter&gt;
      &lt;filter **&gt;
        @type record_transformer
        @id filter_containers_stream_transformer
        &lt;record&gt;
          stream_name ${tag_parts[3]}
        &lt;/record&gt;
      &lt;/filter&gt;
      &lt;match **&gt;
        @type cloudwatch_logs
        @id out_cloudwatch_logs_containers
        region &quot;#{ENV.fetch('AWS_REGION')}&quot;
        log_group_name &quot;/k8s-nest/#{ENV.fetch('AWS_EKS_CLUSTER_NAME')}/containers&quot;
        log_stream_name_key stream_name
        remove_log_stream_name_key true
        auto_create_stream true
        &lt;buffer&gt;
          flush_interval 5
          chunk_limit_size 2m
          queued_chunks_limit_size 32
          retry_forever true
        &lt;/buffer&gt;
      &lt;/match&gt;
    &lt;/label&gt;

Deamonset
==========
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: fluentd-cloudwatch
  name: fluentd-cloudwatch
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-cloudwatch
  template:
    metadata:
      labels:
        k8s-app: fluentd-cloudwatch
      annotations:
        iam.amazonaws.com/role: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
        - env:
            - name: AWS_REGION
              value: us-west-1
            - name: AWS_EKS_CLUSTER_NAME
              value: dex-eks-west
          #image: 'fluent/fluentd-kubernetes-daemonset:v1.1-debian-cloudwatch'
          image: 'fluent/fluentd-kubernetes-daemonset:v1.15.3-debian-cloudwatch-1.1'
          imagePullPolicy: IfNotPresent
          name: fluentd-cloudwatch
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /config-volume
              name: config-volume
            - mountPath: /fluentd/etc
              name: fluentdconf
            - mountPath: /var/log
              name: varlog
            - mountPath: /var/lib/docker/containers
              name: varlibdockercontainers
              readOnly: true
            - mountPath: /run/log/journal
              name: runlogjournal
              readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
        - command:
            - sh
            - '-c'
            - cp /config-volume/..data/* /fluentd/etc
          image: busybox
          imagePullPolicy: Always
          name: copy-fluentd-config
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /config-volume
              name: config-volume
            - mountPath: /fluentd/etc
              name: fluentdconf
      terminationGracePeriodSeconds: 30
      volumes:
        - configMap:
            defaultMode: 420
            name: fluentd-config
          name: config-volume
        - emptyDir: {}
          name: fluentdconf
        - hostPath:
            path: /var/log
            type: ''
          name: varlog
        - hostPath:
            path: /var/lib/docker/containers
            type: ''
          name: varlibdockercontainers
        - hostPath:
            path: /run/log/journal
            type: ''
          name: runlogjournal
</code></pre>

<p>I had the same problem a while ago.</p>
<blockquote>
<p>It seems to be an issue between the logs being emitted from the container and what is being written to the log file. Something is prefixing all logs with the  &lt;stdout/stderr&gt; &lt;?&gt; </p>
</blockquote>
<p>Ref. <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/issues/434#issuecomment-747173567" rel="nofollow noreferrer">https://github.com/fluent/fluentd-kubernetes-daemonset/issues/434#issuecomment-747173567</a></p>
<p>Try following the discussion in the link I pasted you above; I solved it like this:</p>
<pre><code>  &lt;parse&gt;
    @type regexp
    expression /^(?&lt;time&gt;.+) (?&lt;stream&gt;stdout|stderr)( (?&lt;logtag&gt;.))? (?&lt;log&gt;.*)$/
  &lt;/parse&gt;
</code></pre>
