<p>We have configured Fluent-bit to send the logs from our cluster directly to CloudWatch.
We have enabled the Kubernetes filter in order to set our log_stream_name as $(kubernetes['container_name']).</p>
<p>However, the logs are terrible.</p>
<p>Each CloudWatch line looks like this:</p>
<pre><code>    2022-06-23T14:17:34.879+02:00   {&quot;kubernetes&quot;:{&quot;redacted_redacted&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25&quot;,&quot;redacted_image&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45&quot;,&quot;redacted_name&quot;:&quot;redacted-redacted&quot;,&quot;docker_id&quot;:&quot;b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a&quot;,&quot;host&quot;:&quot;ip-0.0.0.0.region-#.compute.internal&quot;,&quot;namespace_name&quot;:&quot;namespace&quot;,&quot;pod_id&quot;:&quot;podpodpod-296c-podpod-8954-podpodpod&quot;,&quot;pod_name&quot;:&quot;redacted-redacted-redacted-7dcbfd4969-mb5f5&quot;},
    2022-06-23T14:17:34.879+02:00   {&quot;kubernetes&quot;:{&quot;redacted_redacted&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25&quot;,&quot;redacted_image&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45&quot;,&quot;redacted_name&quot;:&quot;redacted-redacted&quot;,&quot;docker_id&quot;:&quot;b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a&quot;,&quot;host&quot;:&quot;ip-0.0.0.0.region-#.compute.internal&quot;,&quot;namespace_name&quot;:&quot;namespace&quot;,&quot;pod_id&quot;:&quot;podpodpod-296c-podpod-8954-podpodpod&quot;,&quot;pod_name&quot;:&quot;redacted-redacted-redacted-7dcbfd4969-mb5f5&quot;},
    2022-06-23T14:17:34.879+02:00   {&quot;kubernetes&quot;:{&quot;redacted_redacted&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25&quot;,&quot;redacted_image&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45&quot;,&quot;redacted_name&quot;:&quot;redacted-redacted&quot;,&quot;docker_id&quot;:&quot;b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a&quot;,&quot;host&quot;:&quot;ip-0.0.0.0.region-#.compute.internal&quot;,&quot;namespace_name&quot;:&quot;namespace&quot;,&quot;pod_id&quot;:&quot;podpodpod-296c-podpod-8954-podpodpod&quot;,&quot;pod_name&quot;:&quot;redacted-redacted-redacted-7dcbfd4969-mb5f5&quot;},
    2022-06-23T14:20:07.074+02:00   {&quot;kubernetes&quot;:{&quot;redacted_redacted&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25&quot;,&quot;redacted_image&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45&quot;,&quot;redacted_name&quot;:&quot;redacted-redacted&quot;,&quot;docker_id&quot;:&quot;b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a&quot;,&quot;host&quot;:&quot;ip-0.0.0.0.region-#.compute.internal&quot;,&quot;namespace_name&quot;:&quot;namespace&quot;,&quot;pod_id&quot;:&quot;podpodpod-296c-podpod-8954-podpodpod&quot;,&quot;pod_name&quot;:&quot;redacted-redacted-redacted-7dcbfd4969-mb5f5&quot;},
</code></pre>
<p>Which makes the logs unusable unless expanded, and once expanded the logs look like this:</p>
<pre><code>2022-06-23T14:21:34.207+02:00
{
    &quot;kubernetes&quot;: {
        &quot;container_hash&quot;: &quot;145236632541.lfl.ecr.region.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25&quot;,
        &quot;container_image&quot;: &quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45&quot;,
        &quot;container_name&quot;: &quot;redacted-redacted&quot;,
        &quot;docker_id&quot;: &quot;b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a&quot;,
        &quot;host&quot;: &quot;ip-0.0.0.0.region-#.compute.internal&quot;,
        &quot;namespace_name&quot;: &quot;redacted&quot;,
        &quot;pod_id&quot;: &quot;podpodpod-296c-podpod-8954-podpodpod&quot;,
        &quot;pod_name&quot;: &quot;redacted-redacted-redacted-7dcbfd4969-mb5f5&quot;
    },
    &quot;log&quot;: &quot;[23/06/2022 12:21:34] loglineloglinelogline\ loglineloglinelogline \n&quot;,
    &quot;stream&quot;: &quot;stdout&quot;
}
    {&quot;kubernetes&quot;:{&quot;redacted_redacted&quot;:&quot;145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25&quot;,&quot;redacted_image
</code></pre>
<p>Which is also a bit horrible because every line is flooded with Kubernetes data.
I would like to remove the Kubernetes data from the logs completely,
But I would like to keep using $(kubernetes['container_name']) as the log stream name so that the logs are properly named.
I have tried using filters with Remove_key and LUA scripts that would remove the Kubernetes data. But as soon as something removes it, the log stream cannot be named $(kubernetes['container_name']).</p>
<p>I have found very little documentation on this.
And have not found a proper way to remove Kubernetes data and to keep my log_stream_name as my container_name.</p>
<p>Here is the raw with the fluent bit config that I used:
<a href="https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml" rel="nofollow noreferrer">https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml</a></p>
<p>Any help would be appreciated.</p>

Zhyer
<p>There is an instruction <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html</a> / (Optional) Reducing the log volume from Fluent Bit</p>
<p>Just adding nest filter in the log config. Eg</p>
<pre><code>user-api.conf: |
[INPUT]
    Name                tail
    Tag                 user-api.*
    Path                /var/log/containers/user-api*.log
    Docker_Mode         On
    Docker_Mode_Flush   5
    Docker_Mode_Parser  container_firstline_user
    Parser              docker
    DB                  /var/fluent-bit/state/flb_user_api.db
    Mem_Buf_Limit       50MB
    Skip_Long_Lines     On
    Refresh_Interval    10
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
    Name                kubernetes
    Match               user-api.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     user-api.var.log.containers.
    Merge_Log           On
    Merge_Log_Key       log_processed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off

[FILTER]
    Name                grep
    Match               user-api.*
    Exclude             log /.*&quot;GET \/ping HTTP\/1.1&quot; 200.*/
    
[FILTER]
    Name                nest
    Match               user-api.*
    Operation           lift
    Nested_under        kubernetes
    Add_prefix          Kube.

[FILTER]
    Name                modify
    Match               user-api.*
    Remove              kubernetes.kubernetes.host
    Remove              Kube.container_hash
    Remove              Kube.container_image
    Remove              Kube.container_name
    Remove              Kube.docker_id
    Remove              Kube.host
    Remove              Kube.pod_id

[FILTER]
    Name                nest
    Match               user-api.*
    Operation           nest
    Wildcard            Kube.*
    Nested_under        kubernetes
    Remove_prefix       Kube.

[OUTPUT]
    Name                cloudwatch_logs
    Match               user-api.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/user-api
    log_stream_prefix   app-
    auto_create_group   true
    extra_user_agent    container-insights
</code></pre>
