<p>I'm trying to implement a <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#streaming-sidecar-container" rel="nofollow noreferrer">Streaming Sidecar Container</a> logging architecture in Kubernetes using Fluentd.</p>

<p>In a single pod I have:</p>

<ul>
<li><a href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir" rel="nofollow noreferrer">emptyDir</a> Volume (as log storage)</li>
<li>Application container</li>
<li>Fluent log-forwarder container</li>
</ul>

<p>Basically, the Application container logs are stored in the shared emptyDir volume. Fluentd log-forwarder container tails this log file in the shared emptyDir volume and forwards it an external log-aggregator.</p>

<p>The Fluentd log-forwarder container uses the following config in <code>td-agent.conf</code>:</p>

<pre><code>&lt;source&gt;
  @type tail
  tag "#{ENV['TAG_VALUE']}"
  path (path to log file in volume)
  pos_file /var/log/td-agent/tmp/access.log.pos
  format json
  time_key time
  time_format %iso8601
  keep_time_key true
&lt;/source&gt;

&lt;match *.*&gt;
  @type forward
  @id forward_tail
  heartbeat_type tcp
  &lt;server&gt;
    host (server-host-address)
  &lt;/server&gt;
&lt;/match&gt;
</code></pre>

<p>I'm using an environment variable to set the <code>tag</code> value so I can change it dynamically e.g. when I have to use this container side-by-side with a different Application container, I don't have to modify this config and rebuild this image again.</p>

<p>Now, I set the environment variable value during pod creation in Kubernetes:</p>

<pre><code>    .
    .
    spec:
      containers:
      - name: application-pod
        image: application-image:1.0
        ports:
        - containerPort: 1234
        volumeMounts:
        - name: logvolume
          mountPath: /var/log/app
      - name: log-forwarder
        image: log-forwarder-image:1.0
        env:
        - name: "TAG_VALUE"
          value: "app.service01"
        volumeMounts:
        - name: logvolume
          mountPath: /var/log/app
      volumes:
      - name: logvolume
        emptyDir: {}
</code></pre>

<p>After deploying the pod, I found that the tag value in the Fluentd log-forwarder container comes out empty (expected value: "app.service01"). I imagine it's because Fluentd's td-agent initializes first before the <code>TAG_VALUE</code> environment variable gets assigned.</p>

<p>So, the main question is...<br>
<strong>How can I dynamically set the td-agent's tag value?</strong></p>

<p>But really, what I'm wondering is:<br>
<strong>Is it possible to assign an environment variable before a container's initialization in Kubernetes?</strong></p>

libzz
<p>As an answer to your first question (<strong>How can I dynamically set the td-agent's tag value?</strong>), this seems the best way that you are doing which is defining <code>tag "#{ENV['TAG_VALUE']}"</code> inside fluentd config file.</p>

<p>For your second question, environment variable is assigned before a container's initialization. </p>

<p>So it means it should work and I tested with below sample yaml, and it just worked fine.</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-conf
data:
  fluentd.conf.template: |
    &lt;source&gt;
      @type tail
      tag "#{ENV['TAG_VALUE']}"
      path /var/log/nginx/access.log
      format nginx
    &lt;/source&gt;
    &lt;match *.*&gt;
      @type stdout
    &lt;/match&gt;
---
apiVersion: v1
kind: Pod
metadata:
  name: log-forwarder
  labels:
    purpose: test-fluentd
spec:
  containers:
    - name: nginx
      image: nginx:latest
      volumeMounts:
        - name: logvolume
          mountPath: /var/log/nginx
    - name: fluentd
      image: fluent/fluentd
      env:
        - name: "TAG_VALUE"
          value: "test.nginx"
        - name: "FLUENTD_CONF"
          value: "fluentd.conf"
      volumeMounts:
        - name: fluentd-conf
          mountPath: /fluentd/etc
        - name: logvolume
          mountPath: /var/log/nginx
  volumes:
    - name: fluentd-conf
      configMap:
        name: fluentd-conf
        items:
          - key: fluentd.conf.template
            path: fluentd.conf
    - name: logvolume
      emptyDir: {}
  restartPolicy: Never
</code></pre>

<p>And when I curl nginx pod, I see this output on fluentd containers stdout.</p>

<pre><code>kubectl logs -f log-forwarder fluentd

2019-03-20 09:50:54.000000000 +0000 test.nginx: {"remote":"10.20.14.1","host":"-","user":"-","method":"GET","path":"/","code":"200","size":"612","referer":"-","agent":"curl/7.60.0","http_x_forwarded_for":"-"}
2019-03-20 09:50:55.000000000 +0000 test.nginx: {"remote":"10.20.14.1","host":"-","user":"-","method":"GET","path":"/","code":"200","size":"612","referer":"-","agent":"curl/7.60.0","http_x_forwarded_for":"-"}
2019-03-20 09:50:56.000000000 +0000 test.nginx: {"remote":"10.128.0.26","host":"-","user":"-","method":"GET","path":"/","code":"200","size":"612","referer":"-","agent":"curl/7.60.0","http_x_forwarded_for":"-"}
</code></pre>

<p>As you can see, my environment variable <code>TAG_VALUE=test.nginx</code> has applied to log entries.</p>

<p>I hope it will be useful.</p>
