<h1>AWS CloudWatch Logs in Docker</h1>

<p>Setting an AWS CloudWatch Logs driver in <code>docker</code> is done with <code>log-driver=awslogs</code> and <code>log-opt</code>, for example -</p>

<pre><code>#!/bin/bash

docker run \
    --log-driver=awslogs \
    --log-opt awslogs-region=eu-central-1 \
    --log-opt awslogs-group=whatever-group \
    --log-opt awslogs-stream=whatever-stream \
    --log-opt awslogs-create-group=true \
    wernight/funbox \
        fortune
</code></pre>

<h1>My Problem</h1>

<p>I would like to use AWS CloudWatch logs in a Kubernetes cluster, where each pod contains a few Docker containers. Each deployment would have a separate Log Group, and each container would have a separate stream. I could not find a way to send the logging parameters to the docker containers via Kubernetes <code>create</code> / <code>apply</code>.</p>

<h1>My Question</h1>

<p><strong>How can I send the <code>log-driver</code> and <code>log-opt</code> parameters to a Docker container in a pod / deployment?</strong></p>

<h1>What have I tried</h1>

<ul>
<li>Setting relevant parameters for the Docker daemon on each machine. It's possible, but this way all the containers on the same machine would share the same stream - therefore irrelevant for my case.</li>
<li>RTFM for <a href="https://kubernetes-v1-4.github.io/docs/user-guide/kubectl/kubectl_apply/" rel="noreferrer"><code>kubectl apply</code></a></li>
<li>Reading the <a href="https://github.com/kubernetes/kops/blob/master/vendor/github.com/docker/docker/docs/admin/logging/awslogs.md" rel="noreferrer">relevant README in <code>kops</code></a></li>
<li>Read <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="noreferrer"><code>Kubernetes Logging Architecture</code></a></li>
</ul>

<p>From what I understand, Kubernetes prefer Cluster-level logging to Docker logging driver.</p>

<p>We could use <a href="http://fluentd.org" rel="noreferrer">fluentd</a> to collect, transform, and push container logs to CloudWatch Logs.</p>

<p>All you need is to create a fluentd DaemonSet with ConfigMap and Secret. Files can be found in <a href="https://github.com/zerda/kube-fluentd-cloudwatch" rel="noreferrer">Github</a>. It has been tested with Kubernetes v1.7.5. </p>

<p>The following are some explains.</p>

<h1>In</h1>

<p>With DaemonSet, fluentd collect every container logs from the host folder <code>/var/lib/docker/containers</code>.</p>

<h1>Filter</h1>

<p><a href="https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter" rel="noreferrer">fluent-plugin-kubernetes_metadata_filter</a> plugin load the pod's metadata from Kubernetes API server.</p>

<p>The log record would be like this.</p>

<pre><code>{
    "log": "INFO: 2017/10/02 06:44:13.214543 Discovered remote MAC 62:a1:3d:f6:eb:65 at 62:a1:3d:f6:eb:65(kube-235)\n",
    "stream": "stderr",
    "docker": {
        "container_id": "5b15e87886a7ca5f7ebc73a15aa9091c9c0f880ee2974515749e16710367462c"
    },
    "kubernetes": {
        "container_name": "weave",
        "namespace_name": "kube-system",
        "pod_name": "weave-net-4n4kc",
        "pod_id": "ac4bdfc1-9dc0-11e7-8b62-005056b549b6",
        "labels": {
            "controller-revision-hash": "2720543195",
            "name": "weave-net",
            "pod-template-generation": "1"
        },
        "host": "kube-234",
        "master_url": "https://10.96.0.1:443/api"
    }
}
</code></pre>

<p>Make some tags with Fluentd <a href="https://github.com/zerda/kube-fluentd-cloudwatch/blob/master/fluentd-configmap.yaml#L27-L28" rel="noreferrer">record_transformer</a> filter plugin.</p>

<pre><code>{
    "log": "...",
    "stream": "stderr",
    "docker": {
        ...
    },
    "kubernetes": {
        ...
    },
    "pod_name": "weave-net-4n4kc",
    "container_name": "weave"
}
</code></pre>

<h1>Out</h1>

<p><a href="https://github.com/ryotarai/fluent-plugin-cloudwatch-logs" rel="noreferrer">fluent-plugin-cloudwatch-logs</a> plugin send to AWS CloudWatch Logs.</p>

<p>With <code>log_group_name_key</code> and <code>log_stream_name_key</code> configuration, log group and stream name can be any field of the record.</p>

<pre><code>&lt;match kubernetes.**&gt;
  @type cloudwatch_logs
  log_group_name_key pod_name
  log_stream_name_key container_name
  auto_create_stream true
  put_log_events_retry_limit 20
&lt;/match&gt;
</code></pre>
