<p>We have 2 clusters on <strong>GKE</strong>: <code>dev</code> and <code>production</code>. I tried to run this command on <code>dev</code> cluster:</p>

<pre><code>gcloud beta container clusters update "dev" --update-addons=NodeLocalDNS=ENABLED 
</code></pre>

<p>And everything went great, node-local-dns pods are running and all works, next morning I decided to run same command on <code>production</code> cluster and node-local-dns fails to run, and I noticed that both <strong>PILLAR__LOCAL__DNS</strong> and <strong>PILLAR__DNS__SERVER</strong> in yaml aren't changed to proper IPs, I tried to change those variables in config yaml, but <strong>GKE</strong> keeps overwriting them back to yaml with <strong>PILLAR__DNS__SERVER</strong> variables... </p>

<p>The only difference between clusters is that <code>dev</code> runs on <code>1.15.9-gke.24</code> and production <code>1.15.11-gke.1</code>.</p>

Kikun
<p><strong>Apparently <code>1.15.11-gke.1</code> version has a bug.</strong></p>

<p>I recreated it first on <code>1.15.11-gke.1</code> and can confirm that <code>node-local-dns</code> <code>Pods</code> fall into <code>CrashLoopBackOff</code> state:</p>

<pre><code>node-local-dns-28xxt                                        0/1     CrashLoopBackOff   5          5m9s
node-local-dns-msn9s                                        0/1     CrashLoopBackOff   6          8m17s
node-local-dns-z2jlz                                        0/1     CrashLoopBackOff   6          10m
</code></pre>

<p>When I checked the logs:</p>

<pre><code>$ kubectl logs -n kube-system node-local-dns-msn9s
2020/04/07 21:01:52 [FATAL] Error parsing flags - Invalid localip specified - "__PILLAR__LOCAL__DNS__", Exiting
</code></pre>

<h3>Solution:</h3>

<p><strong>Upgrade to <code>1.15.11-gke.3</code> helped.</strong> First you need to upgrade your <strong>master-node</strong> and then your <strong>node pool</strong>. It looks like on this version everything runs nice and smoothly:</p>

<pre><code>$ kubectl get daemonsets -n kube-system node-local-dns 
NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                               AGE
node-local-dns   3         3         3       3            3           addon.gke.io/node-local-dns-ds-ready=true   44m

$ kubectl get pods -n kube-system -l k8s-app=node-local-dns
NAME                   READY   STATUS    RESTARTS   AGE
node-local-dns-8pjr5   1/1     Running   0          11m
node-local-dns-tmx75   1/1     Running   0          19m
node-local-dns-zcjzt   1/1     Running   0          19m
</code></pre>

<p>As it comes to manually fixing this particular daemonset <code>yaml</code> file, I wouldn't recommend it as you can be sure that <strong>GKE's</strong> <strong>auto-repair</strong> and <strong>auto-upgrade</strong> features will overwrite it sooner or later anyway.</p>

<p>I hope it was helpful.</p>
