<p>I'm trying to collect my application containers' logs throughout their entire life cycle. These containers are running inside Kubernetes pods, I've found solutions like Fluentd but I also found out that I need to specify a backend (Elasticsearch, AWS S3, etc.) whereas I want to collect logs inside files having specific names, for example podname_namespace_containername.json and then parse those files using a script. Is this possible with fluentd?</p>

<p>By far the fastest way to setup log collection is <a href="https://github.com/helm/charts/tree/master/stable/fluent-bit" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/fluent-bit</a>. Refer <code>values.yaml</code> for all the options available. It supports multiple backends like ES, S3, Kafka. Every log event is enriched with pod metadata (pod name, namespace, etc) and tagged so that you can organize processing separately on a backend. E.g. on a backend you can select and parse only certain pods in certain namespaces.</p>
<p>According to <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/cluster-administration/logging/</a> you log to stdout/stderr, it gets written to the underlying node, a log collector (daemonset) collects everything and sends further.</p>
<p><a href="https://i.stack.imgur.com/J4h3p.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/J4h3p.png" alt="enter image description here" /></a></p>
<p>FluentBit daemonset in Kubernetes implements exactly this architecture. More docs on FluentBit: <a href="https://docs.fluentbit.io/manual/concepts/data-pipeline" rel="nofollow noreferrer">https://docs.fluentbit.io/manual/concepts/data-pipeline</a></p>
