<p>I'm trying to capture some logs that are file-based in an application pod on GKE and view them from Google Cloud Logging.</p>
<p>For various reasons, these application logs are not sent to STDOUT or STDERR (since those logs are automatically sent to Cloud Logging). I have been suggested to implement a scripting solution that tails the logs and sends them to STDOUT. However I was hoping in a side-car approach with Fluentd (or Fluentbit) logging agent that'll tail the logs and output them to Cloud Logging.</p>
<p>Using the sidecar image <code>&quot;k8s.gcr.io/fluentd-gcp:1.30&quot;</code>, I've tried out the below YAML file (containing the fluentd configmap and deployment) below:</p>
<pre><code>---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-log-config
data:
  fluentd.conf: |
    &lt;source&gt;
      type tail
      format none
      path /var/log/execution*.log
      pos_file /var/log/execution.pos
      tag app.*
    &lt;/source&gt;

    &lt;match **&gt;
      type google_cloud
    &lt;/match&gt;
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app.kubernetes.io/name: app
    app.kubernetes.io/instance: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: app
      app.kubernetes.io/instance: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
        app.kubernetes.io/instance: app
    spec:
      serviceAccountName: app
      volumes:
        - name: executionlogs
          emptyDir: {}
        - name: fluentdconfig
          configMap:
            name: app-log-config
      containers:
        - name: app
          image: appimage:version
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: executionlogs
              mountPath: /tmp/executionLogs
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
        - name: log-agent
          image: &quot;k8s.gcr.io/fluentd-gcp:1.30&quot;
          imagePullPolicy: IfNotPresent
          env:
            - name: FLUENTD_ARGS
              value: &quot;-c /etc/fluentd-config/fluentd.conf&quot;
          volumeMounts:
            - name: executionlogs
              mountPath: /var/log
            - name: fluentdconfig
              mountPath: /etc/fluentd-config
</code></pre>
<p>Initially, the sidecar logs were throwing a 403 error, since I hadn't given the service account the requisite permissions (I was using GKE workload identity and the corresponding GCP IAM service account required logWriter permissions to be added). After fixing the error, I got the following logs:</p>
<pre><code>2021-06-27 12:49:09 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path=&quot;/etc/fluentd-config/fluentd.conf&quot;
2021-06-27 12:49:09 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.29
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-google-cloud' version '0.5.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-kafka' version '0.3.1'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-mongo' version '0.7.15'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-record-reformer' version '0.8.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-s3' version '0.7.1'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-systemd' version '0.0.5'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td' version '0.10.29'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.29'
2021-06-27 12:49:09 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern=&quot;**&quot; type=&quot;google_cloud&quot;
2021-06-27 12:49:10 +0000 [info]: plugin/out_google_cloud.rb:519:block in detect_platform: Detected GCE platform
2021-06-27 12:49:10 +0000 [info]: plugin/out_google_cloud.rb:290:configure: Logs viewer address: https://console.developers.google.com/
project/projectname/logs?service=compute.googleapis.com&amp;key1=instance&amp;key2=9071465168741286442
2021-06-27 12:49:10 +0000 [info]: fluent/root_agent.rb:147:add_source: adding source type=&quot;tail&quot;
2021-06-27 12:49:10 +0000 [info]: fluent/engine.rb:133:configure: using configuration file: &lt;ROOT&gt;
  &lt;source&gt;
    type tail
    format none
    path /var/log/execution*.log
    pos_file /var/log/execution.pos
    tag app.*
  &lt;/source&gt;
  &lt;match **&gt;
    type google_cloud
  &lt;/match&gt;
&lt;/ROOT&gt;
2021-06-27 12:52:10 +0000 [info]: plugin/in_tail.rb:557:initialize: following tail of /var/log/execution1.log
2021-06-27 12:53:10 +0000 [info]: plugin/out_google_cloud.rb:451:block in write: Successfully sent to Google Cloud Logging API.
</code></pre>
<p>Despite the successful message, I don't see anything on the Cloud Logging end.</p>
<p>So, here are my questions:</p>
<ol>
<li>Is this there a better solution for my use-case?</li>
<li>Is the sidecar image I should be using? I wasn't able to find any other fluentd images and the one I'm using is 3 years old. I'd prefer to use something recommended by Google rather than create my own.</li>
<li>What do I additionally need to do to see the logs on Cloud Logging? How do I debug this further?</li>
</ol>
<p>Thanks!</p>

Sayon Roy Choudhury
<p>I’ve tried implementing the configuration you’ve implemented and faced the same issue. I then configured all sources to stream output to STDOUT and was able to view logs on the Cloud Logging dashboard.</p>
<p>Below are the sample configurations I’ve used.</p>
<p>Sample_map-config.yaml:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    &lt;source&gt;
      type tail
      format none
      path /var/log/1.log
      pos_file /var/log/1.log.pos
      tag count.format1
    &lt;/source&gt;

    &lt;source&gt;
      type tail
      format none
      path /var/log/2.log
      pos_file /var/log/2.log.pos
      tag count.format2
    &lt;/source&gt;

    &lt;match **&gt;
      type stdout
    &lt;/match&gt; 
</code></pre>
<p>Sample-pod.yaml:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - &gt;
      i=0;
      while true;
      do
        echo &quot;$i: $(date)&quot; &gt;&gt; /var/log/1.log;
        echo &quot;$(date) INFO $i&quot; &gt;&gt; /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done      
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-agent
    image: k8s.gcr.io/fluentd-gcp:1.30
    env:
    - name: FLUENTD_ARGS
      value: -c /etc/fluentd-config/fluentd.conf
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: config-volume
      mountPath: /etc/fluentd-config
  volumes:
  - name: varlog
    emptyDir: {}
  - name: config-volume
    configMap:
      name: fluentd-config
</code></pre>
