<p>I have a strange error with kube-dns.</p>

<p>Environment:
Cluster with a single master and a few nodes deployed on AWS with kops.
Kubernetes version 1.8.4.</p>

<p>Problem is that I have flakiness in DNS name resolution (both cluster-internal or external names) in my pods. After troubleshooting I understood the problem arises only when pods are scheduled on a specific node, which is the one where one of the replicas of the kube-dns pod is running.</p>

<p>These are my kube-dns pods:</p>

<pre><code>$ kubectl -n kube-system get po -l k8s-app=kube-dns -o wide
NAME                        READY     STATUS    RESTARTS   AGE       IP             NODE
kube-dns-7f56f9f8c7-2ztbn   3/3       Running   0          2d        100.96.8.239   node01
kube-dns-7f56f9f8c7-h5w29   3/3       Running   0          17d       100.96.7.114   node02
</code></pre>

<p>If I run a test POD forcing it to run on <code>node02</code> everything seems fine. I can resolve any (valid) DNS name with no issues at all.</p>

<p>If I run the same test POD on <code>node01</code> name resolution is flaky: sometimes it fails (roughly 50% of the times) with the following error</p>

<pre><code>$ dig google.com
;; reply from unexpected source: 100.96.8.239#53, expected 100.64.0.10#53
</code></pre>

<p>The rest of the times it works flawlessly:</p>

<pre><code>$ dig google.com

; &lt;&lt;&gt;&gt; DiG 9.10.4-P3 &lt;&lt;&gt;&gt; google.com
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 24454
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             60      IN      A       209.85.202.100
google.com.             60      IN      A       209.85.202.101
google.com.             60      IN      A       209.85.202.102
google.com.             60      IN      A       209.85.202.113
google.com.             60      IN      A       209.85.202.138
google.com.             60      IN      A       209.85.202.139

;; Query time: 2 msec
;; SERVER: 100.64.0.10#53(100.64.0.10)
;; WHEN: Mon Jan 08 10:46:42 UTC 2018
;; MSG SIZE  rcvd: 135
</code></pre>

<p><code>/etc/resolv.conf</code> points correctly to the <code>kube-dns</code> service's IP address:</p>

<pre><code>$ head -n 1 /etc/resolv.conf 
nameserver 100.64.0.10

$ kubectl -n kube-system get svc kube-dns 
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   100.64.0.10   &lt;none&gt;        53/UDP,53/TCP   33d
</code></pre>

<p>Apparently, on <code>node01</code> only, when the request is forwarded by the service to the <code>kube-dns</code> pod instance that runs on <code>node01</code> itself the error is triggered.</p>

<p>I tried restarting kube-proxy on <code>node01</code> but the problem remains.</p>

<p>I bet that rebooting/recreating <code>node01</code> would make the problem go away, but I need to ensure this problem doesn't happen again.</p>

<p>Does anybody have an idea what's going on?</p>

<p>I found an <a href="https://github.com/kubernetes/kubernetes/issues/21613" rel="noreferrer">issue on github</a> that looks very similar to this one I am having, and the <a href="https://github.com/kubernetes/kubernetes/issues/21613#issuecomment-343190401" rel="noreferrer">solution</a> posted there seems to work.</p>

<p>Basically, I needed to load a kernel module with the following command:</p>

<pre><code>modprobe br_netfilter
</code></pre>

<p>Of course, YMMV</p>
