<p>we recently updated our AKS cluster from 1.17.x to 1.19.x and recognised that the format of our custom application logs in <code>/var/lib/docker/containers</code> changed.</p>
<p>Before the update it looked like this:
<a href="https://i.stack.imgur.com/0GJio.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0GJio.png" alt="old valid json format" /></a></p>
<p>Afterwards it looks like this:
<a href="https://i.stack.imgur.com/PW1LN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PW1LN.png" alt="new invalid json format" /></a></p>
<p>I can find some notes in the changelog that kubernetes changed from just text logs to structured logs (for system components) but I don't see how this correlates to how our log format changed.</p>
<p><a href="https://kubernetes.io/blog/2020/09/04/kubernetes-1-19-introducing-structured-logs/#:%7E:text=In%20Kubernetes%201.19%2C%20we%20are,migrated%20to%20the%20structured%20format" rel="nofollow noreferrer">https://kubernetes.io/blog/2020/09/04/kubernetes-1-19-introducing-structured-logs/#:~:text=In%20Kubernetes%201.19%2C%20we%20are,migrated%20to%20the%20structured%20format</a></p>
<p><a href="https://kubernetes.io/docs/concepts/cluster-administration/system-logs/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/cluster-administration/system-logs/</a></p>
<p>Is there a chance to still get valid json logs to <code>/var/lib/docker/containers</code> in AKS &gt; 1.19.x?</p>
<p>Background:
We send our application logs to Splunk and don't use the Azure stack for log analysis. Our Splunk setup cannot parse that new log format as of now.</p>

<p>The format of the logs are defined by the container runtime. It seems before you were parsing logs from docker container runtime, and now it is containerd (<a href="https://azure.microsoft.com/en-us/updates/azure-kubernetes-service-aks-support-for-containerd-runtime-is-in-preview/" rel="noreferrer">https://azure.microsoft.com/en-us/updates/azure-kubernetes-service-aks-support-for-containerd-runtime-is-in-preview/</a>).</p>
<p>Based on the article - you can still choose moby (which is docker) as the container runtime.</p>
<p>To take that also from your shoulders, you should look into using one of those (considering that they will automatically detect the log format and container runtime for you).</p>
<ul>
<li><p>Splunk Connect for Kubernetes <a href="https://github.com/splunk/splunk-connect-for-kubernetes" rel="noreferrer">https://github.com/splunk/splunk-connect-for-kubernetes</a></p>
</li>
<li><p>Collectord <a href="https://www.outcoldsolutions.com" rel="noreferrer">https://www.outcoldsolutions.com</a></p>
</li>
</ul>
