<p>I would like to forward Kubernetes logs from fluent-bit to elasticsearch through fluentd but fluent-bit cannot parse kubernetes logs properly. In order to install Fluent-bit and Fluentd, I use Helm charts. I tried both stable/fluentbit and fluent/fluentbit and faced with same problem:</p>
<pre><code>#0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=&quot;400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'Could not dynamically add mapping for field [app.kubernetes.io/component]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].'&quot;
</code></pre>
<p>I put following lines into fluent-bit values file as shown <a href="https://github.com/kruftik/fluent-helm-charts/blob/feature/dedot-metadata-keys/charts/fluent-bit/values.yaml" rel="nofollow noreferrer">here</a></p>
<pre><code>  remapMetadataKeysFilter:
    enabled: true
    match: kube.*

    ## List of the respective patterns and replacements for metadata keys replacements
    ## Pattern must satisfy the Lua spec (see https://www.lua.org/pil/20.2.html)
    ## Replacement is a plain symbol to replace with
    replaceMap:
      - pattern: &quot;[/.]&quot;
        replacement: &quot;_&quot;
</code></pre>
<p>...nothing changed, same errors are listed.</p>
<p>Is there a workaround to get rid of that bug?</p>
<p>my values.yaml is here:</p>
<pre><code>    # Default values for fluent-bit.

# kind -- DaemonSet or Deployment
kind: DaemonSet

# replicaCount -- Only applicable if kind=Deployment
replicaCount: 1

image:
  repository: fluent/fluent-bit
  pullPolicy: Always
  # tag:

imagePullSecrets: []
nameOverride: &quot;&quot;
fullnameOverride: &quot;&quot;

serviceAccount:
  create: true
  annotations: {}
  name:

rbac:
  create: true

podSecurityPolicy:
  create: false

podSecurityContext:
  {}
  # fsGroup: 2000

securityContext:
  {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 2020
  annotations:
    prometheus.io/path: &quot;/api/v1/metrics/prometheus&quot;
    prometheus.io/port: &quot;2020&quot;
    prometheus.io/scrape: &quot;true&quot;

serviceMonitor:
  enabled: true
  namespace: monitoring
  interval: 10s
  scrapeTimeout: 10s
  # selector:
  #  prometheus: my-prometheus

resources:
  {}
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

podAnnotations: {}

priorityClassName: &quot;&quot;

env: []

envFrom: []

extraPorts: []
#   - port: 5170
#     containerPort: 5170
#     protocol: TCP
#     name: tcp

extraVolumes: []

extraVolumeMounts: []

## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit
config:
  ## https://docs.fluentbit.io/manual/service
  service: |
    [SERVICE]
        Flush 1
        Daemon Off
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [FILTER]
        Name    lua
        Match   kube.*
        script  /fluent-bit/etc/functions.lua
        call    dedot
        
  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name          forward
        Match         *
        Host          fluentd-in-forward.elastic-system.svc.cluster.local
        Port          24224
        tls           off
        tls.verify    off

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
</code></pre>

<p>I had the same issue, which is caused by multiple labels that are converted into json.
I renamed the conflicting keys to match with the newer format of recommended labels:</p>
<pre><code>&lt;filter **&gt;
  @type rename_key
  rename_rule1 ^app$ app.kubernetes.io/name
  rename_rule2 ^chart$ helm.sh/chart
  rename_rule3 ^version$ app.kubernetes.io/version
  rename_rule4 ^component$ app.kubernetes.io/component
  rename_rule5 ^istio$ istio.io/name
&lt;/filter&gt;
</code></pre>
