<p>We have a GKE cluster with:</p>

<ul>
<li>master nodes with version 1.6.13-gke.0</li>
<li>2 node pools with version 1.6.11-gke.0</li>
</ul>

<p>We have Stackdriver Monitoring and Logging activated.</p>

<p>On 2018-01-22, masters where upgraded by Google to version 1.7.11-gke.1.</p>

<p>After this upgrade, we have a lot of errors like these:</p>

<pre><code>I  2018-01-25 11:35:23 +0000 [error]: Exception emitting record: No such file or directory @ sys_fail2 - (/var/log/fluentd-buffers/kubernetes.system.buffer..b5638802e3e04e72f.log, /var/log/fluentd-buffers/kubernetes.system.buffer..q5638802e3e04e72f.log)

I  2018-01-25 11:35:23 +0000 [warn]: emit transaction failed: error_class=Errno::ENOENT error="No such file or directory @ sys_fail2 - (/var/log/fluentd-buffers/kubernetes.system.buffer..b5638802e3e04e72f.log, /var/log/fluentd-buffers/kubernetes.system.buffer..q5638802e3e04e72f.log)" tag="docker"

I    2018-01-25 11:35:23 +0000 [warn]: suppressed same stacktrace
</code></pre>

<p>Those messages are flooding our logs ~ 25Gb of logs each day, and are generated by pods managed by a DaemonSet called fluentd-gcp-v2.0.9 .</p>

<p>We found that it's a <a href="https://github.com/kubernetes/kubernetes/issues/56653" rel="nofollow noreferrer">bug</a> fixed on 1.8 and <a href="https://github.com/kubernetes/kubernetes/pull/57048" rel="nofollow noreferrer">backported to 1.7.12</a>.</p>

<p>My questions are:</p>

<ol>
<li>Should we upgrade masters to version 1.7.12 ? Is it safe to do it? OR</li>
<li>Is there any other alternative to test before upgrading?</li>
</ol>

<p>Thanks in advance.</p>

<p>First of all, the answer to question 2.</p>

<p>As alternatives we could have:</p>

<ul>
<li>filtered fluentd to ignore logs from fluentd-gcp pods OR</li>
<li>deactivate Stackdriver monitoring and logging</li>
</ul>

<p>To answer question 1:</p>

<p>We upgraded to 1.7.12 in a <strong>test environment</strong>. The process took 3 minutes. During this period of time, we could not edit our cluster nor access it with kubectl (as expected).</p>

<p>After the upgrade, we <strong>deleted</strong> all our pods called <strong>fluentd-gcp-*</strong> and the flood stopped instantly:</p>

<pre><code>for pod in $(kubectl get pods -nkube-system | grep fluentd-gcp | awk '{print $1}'); do \
    kubectl -nkube-system delete pod $pod; \
    sleep 20; \
done;
</code></pre>
