<p dir="auto">Currently we collect the JSON Docker container logs and ingest them without further processing into either Elasticsearch or Cloud Logging. It would be useful to add fields to identify the name of the pod that corresponds to a specific Docker container log file (line) and perhaps also the hostname too. Relating the Docker container ID to a pod name might be possible by speaking to the kubelet during log ingestion time and this would require us to write a suitable filter in Ruby as part of our Flueuntd setup. However, we also need access to historical information because we may need to tag a log file for a Docker container for which the pod is not currently running -- perhaps by looking at events? This then also impacts one of the decisions we are trying to make about where events should be stored -- esp. if we decided to also store them as log files ingested by the same flow.</p>
<p dir="auto">This would then allow users to query "all logs for a pod" and enable the query "all logs for a service".</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/a-robinson/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/a-robinson">@a-robinson</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dchen1107/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dchen1107">@dchen1107</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lavalamp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lavalamp">@lavalamp</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thockin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thockin">@thockin</a></p>
<p dir="auto">All of the labels attached to the container whose logs are being collected should be added on to the logs. For example, if logs are being collected for container foo with the labels env=prod and service=backend, those labels (env=prod, service=backend) should be attached such that someone querying logs later can ask for all logs with those labels to easily aggregate across containers.</p>
<p dir="auto">IIRC both elasticsearch and Google Cloud Logging support arbitrary labels, so the only work needed should be getting the container's labels into Fluentd and attaching them.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satnam6502/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satnam6502">@satnam6502</a></p>