<p>I am a new DevOps Engineer.</p>
<p>I was checking out our DEV AKS cluster at work and noticed that Fluentd is using a crazy amount of memory and isn't releasing it back, example below:</p>
<pre><code>fluentd-dev-95qmh                              13m          1719Mi
fluentd-dev-fhd4w                              9m           1732Mi
fluentd-dev-n22hf                              11m          660Mi
fluentd-dev-qlzd8                              12m          524Mi
fluentd-dev-rg9gp                              9m           2338Mi
</code></pre>
<p>Fluentd is deployed as a daemonset so I can't just scale it up or down, unfortunately.</p>
<p>The version we are running is 1.2.22075.8 and it get deployed via CI/CD pipelines using a deployment.yml file and a dockerfile.</p>
<p>Here is the Dockerfile:</p>
<pre><code>FROM quay.io/fluentd_elasticsearch/fluentd:v3.2.0
#RUN adduser --uid 10000 --gecos '' --disabled-password fluent --no-create-home &amp;&amp; \
    #chown fluent:fluent /entrypoint.sh &amp;&amp; \
    #chown -R fluent:fluent /etc/fluent/ &amp;&amp; \
    #chown -R fluent:fluent /usr/local/bin/ruby &amp;&amp; \
    #chown -R fluent:fluent /usr/local/bundle/bin/fluent* &amp;&amp; \
    #chmod -R fluent:fluent /var/lib/docker/containers &amp;&amp; \
    #chmod -R fluent:fluent /var/log
#USER fluent
</code></pre>
<p>I went to <a href="https://quay.io/repository/fluentd_elasticsearch/fluentd?tab=tags&amp;tag=latest" rel="nofollow noreferrer">https://quay.io/repository/fluentd_elasticsearch/fluentd?tab=tags&amp;tag=latest</a> and saw that there were newer versions available. I wanted to update Fluentd to v3.3.0 and I thought I could just do this by changing the version number in the Dockerfile and triggering a build. I did this the release pipeline failed, two pods were in &quot;CrashLoopBackOff&quot; state and three pods were running normally. I also had a bunch of errors related to Ruby. I know, I should have taken note of the errors but since this was at work I just scared and reverted the version in the Dockerfile back to v3.2.0 from v3.3.0 and triggered a build and everything went back to how it was before.</p>
<p>How do I update the version of the Fluentd daemonset? Is there a way I can restart these pods and clear the memory? I've Googled this question and it doesn't seem like there is a way to do this easily because it is not a regular deployment.</p>
<p>Also, any idea why fluentd would be eating so much memory?</p>
<p>This issue is having a negative impact on the DEV cluster, 3 out of 5 nodes are above 110% memory usage.</p>

ahsm
<p>Regarding the question about restarting the fluentd pods:</p>
<p>If you have permissions to delete pods in the namespace where fluentd is deployed, you can simply delete the pods to restart fluentd</p>
<pre><code>kubectl delete pod fluentd-xxxxx
</code></pre>
<p>Since the daemonset definition is still there the Kubernetes control plane will notice there are fluentd pods missing and start new ones. This will however mean that there will be a short time windows where there will be no fluentd running on the node (i.e the time between the delete command is issued and until the new fluentd pod is operational). The Control plane will detect that there are fluentd pods missing almost instantly, but they do take some time to start.</p>
