<p>I am using AWS Opensearch to retrieve the logs from all my Kubernetes applications.
I have the following pods: <code>Kube-proxy</code>, <code>Fluent-bit</code>, <code>aws-node</code>, <code>aws-load-balancer-controller</code>, and all my apps (around 10).</p>
<p>While fluent-bit successfully send all the logs from <code>Kube-proxy</code>, <code>Fluent-bit</code>, <code>aws-node</code> and <code>aws-load-balancer-controller</code>, none of the logs from my applications are sent. My applications had <code>DEBUG</code>, <code>INFO</code>, <code>ERROR</code> logs, and none are sent by fluent bit.</p>
<p>Here is my fluent bit configuration:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: my-namespace
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            my-host.es.amazonaws.com
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ap-southeast-1
        Retry_Limit     6

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?&lt;host&gt;[^ ]*) [^ ]* (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+)(?: +(?&lt;path&gt;[^\&quot;]*?)(?: +\S*)?)?&quot; (?&lt;code&gt;[^ ]*) (?&lt;size&gt;[^ ]*)(?: &quot;(?&lt;referer&gt;[^\&quot;]*)&quot; &quot;(?&lt;agent&gt;[^\&quot;]*)&quot;)?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?&lt;host&gt;[^ ]*) [^ ]* (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+)(?: +(?&lt;path&gt;[^ ]*) +\S*)?&quot; (?&lt;code&gt;[^ ]*) (?&lt;size&gt;[^ ]*)(?: &quot;(?&lt;referer&gt;[^\&quot;]*)&quot; &quot;(?&lt;agent&gt;[^\&quot;]*)&quot;)?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?&lt;time&gt;[^\]]*)\] \[(?&lt;level&gt;[^\]]*)\](?: \[pid (?&lt;pid&gt;[^\]]*)\])?( \[client (?&lt;client&gt;[^\]]*)\])? (?&lt;message&gt;.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?&lt;remote&gt;[^ ]*) (?&lt;host&gt;[^ ]*) (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+)(?: +(?&lt;path&gt;[^\&quot;]*?)(?: +\S*)?)?&quot; (?&lt;code&gt;[^ ]*) (?&lt;size&gt;[^ ]*)(?: &quot;(?&lt;referer&gt;[^\&quot;]*)&quot; &quot;(?&lt;agent&gt;[^\&quot;]*)&quot;)?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\&lt;(?&lt;pri&gt;[0-9]+)\&gt;(?&lt;time&gt;[^ ]* {1,2}[^ ]* [^ ]*) (?&lt;host&gt;[^ ]*) (?&lt;ident&gt;[a-zA-Z0-9_\/\.\-]*)(?:\[(?&lt;pid&gt;[0-9]+)\])?(?:[^\:]*\:)? *(?&lt;message&gt;.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
</code></pre>
<p>I followed <a href="https://www.eksworkshop.com/intermediate/230_logging/deploy/" rel="nofollow noreferrer">this documentation</a></p>
<p>Thanks a lot for the help.</p>

nolwww
<p>Finally, I did two things that solved my issue:</p>
<ol>
<li>Modified this configuration:</li>
</ol>
<pre><code># before
 output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            search-blacaz-logs-szzq6vokwwm4y5fkfwyngjwjxq.ap-southeast-1.es.amazonaws.com
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ap-southeast-1
        Retry_Limit     6

# after
 output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            search-blacaz-logs-szzq6vokwwm4y5fkfwyngjwjxq.ap-southeast-1.es.amazonaws.com
        Port            443
        TLS             On
        AWS_Auth        On
        Replace_Dots    On // added this
        AWS_Region      ap-southeast-1
        Retry_Limit     6
</code></pre>
<p>Then, I had to delete the fluent-bit Elastic search index, and re-create it.
Indeed, the index was probably not well suited for my JAVA logs at first, and adjusted to it after re-creation.</p>
