<p>I have created a GKE cluster on GCP.</p>
<p>Kubernetes logs from kubectl logs command is different to /var/log/containers</p>
<p>kubectl</p>
<pre><code>{&quot;method&quot;:&quot;GET&quot;,&quot;path&quot;:&quot;/healthz&quot;,&quot;format&quot;:&quot;*/*&quot;,&quot;controller&quot;:&quot;Public::PublicPagesController&quot;,&quot;action&quot;:&quot;healthz&quot;,&quot;status&quot;:204,&quot;duration&quot;:0.39,&quot;view&quot;:0.0,&quot;request_id&quot;:&quot;ca29b519-d1e8-49a2-95ae-e5f23b60c36f&quot;,&quot;params&quot;:{},&quot;custom&quot;:null,&quot;request_time&quot;:&quot;2022-04-27T15:25:43.780+00:00&quot;,&quot;process_id&quot;:6,&quot;@version&quot;:&quot;vcam-backend-vvcam-72_shareholder_event-rc16&quot;,&quot;@timestamp&quot;:&quot;2022-04-27T15:25:43.780Z&quot;,&quot;message&quot;:&quot;[204] GET /healthz (Public::PublicPagesController#healthz)&quot;}
</code></pre>
<p>And logs in /var/log/containers, something add timestamp into the beginning of my container logs:</p>
<pre><code>2022-04-27T15:25:43.780523421Z stdout F {&quot;method&quot;:&quot;GET&quot;,&quot;path&quot;:&quot;/healthz&quot;,&quot;format&quot;:&quot;*/*&quot;,&quot;controller&quot;:&quot;Public::PublicPagesController&quot;,&quot;action&quot;:&quot;healthz&quot;,&quot;status&quot;:204,&quot;duration&quot;:0.39,&quot;view&quot;:0.0,&quot;request_id&quot;:&quot;ca29b519-d1e8-49a2-95ae-e5f23b60c36f&quot;,&quot;params&quot;:{},&quot;custom&quot;:null,&quot;request_time&quot;:&quot;2022-04-27T15:25:43.780+00:00&quot;,&quot;process_id&quot;:6,&quot;@version&quot;:&quot;vcam-backend-vvcam-72_shareholder_event-rc16&quot;,&quot;@timestamp&quot;:&quot;2022-04-27T15:25:43.780Z&quot;,&quot;message&quot;:&quot;[204] GET /healthz (Public::PublicPagesController#healthz)&quot;}
</code></pre>
<p>I want my application log will be consistent, I want it in json format like logs from kubectl command, so I can parse and analyze more.</p>
<p>I want to remove this part: <code>2022-04-27T15:25:43.780523421Z stdout F</code></p>
<p>Does anybody meet this problem? How can I make containers logs same as kubectl command logs?</p>
<p>GKE Version:</p>
<pre><code>Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21&quot;, GitVersion:&quot;v1.21.10-gke.2000&quot;, GitCommit:&quot;0823380786b063c3f71d5e7c76826a972e30550d&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2022-03-17T09:22:22Z&quot;, GoVersion:&quot;go1.16.14b7&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>Docker daemon.json</p>
<pre><code>{
  &quot;pidfile&quot;: &quot;/var/run/docker.pid&quot;,
  &quot;iptables&quot;: false,
  &quot;ip-masq&quot;: false,
  &quot;log-level&quot;: &quot;warn&quot;,
  &quot;bip&quot;: &quot;169.254.123.1/24&quot;,
  &quot;mtu&quot;: 1460,
  &quot;storage-driver&quot;: &quot;overlay2&quot;,
  &quot;live-restore&quot;: true,
  &quot;log-driver&quot;: &quot;json-file&quot;,
  &quot;log-opts&quot;: {
      &quot;max-size&quot;: &quot;10m&quot;,
      &quot;max-file&quot;: &quot;5&quot;
    }
}
</code></pre>
<p>Notes: I notice that the timestamp in the beginning of log line only show when we add option <code>docker logs -t</code>, <a href="https://docs.docker.com/engine/reference/commandline/logs/" rel="nofollow noreferrer">docs here</a>
But I still do not know how to fix this problem in GKE cluster.</p>

Tho Quach
<p>This problem is related to Container Runtime Interface (CRI). You can read about CRI <a href="https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/" rel="nofollow noreferrer">here</a>.</p>
<p>For now, I'm still can not change the log format as I want, I just adapt to this new format. This format is called <code>CRI log format</code>, the CRI default of GKE cluster always produce log in this format, and aggregation log applications adapt to this new CRI log format too:</p>
<ul>
<li>This is PR of <code>Grafana/Loki</code> to support CRI log format: <a href="https://github.com/grafana/loki/pull/365" rel="nofollow noreferrer">PR</a></li>
<li>Fluent-bit create a new parser for CRI log format: <a href="https://docs.fluentbit.io/manual/v/1.8/installation/kubernetes#container-runtime-interface-cri-parser" rel="nofollow noreferrer">Docs</a></li>
</ul>
<p>So I think you need to change the way you approach this problem: If we can not change the log format as we want, we can use application support this log format.</p>
<p>Notes: I'm not sure but I think this problem come from: <code>Kubernetes is removing support for Docker as a container runtime</code> - <a href="https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/" rel="nofollow noreferrer">docs</a> , and the new container runtime produce this CRI log format.</p>
