<p>I'm struggling to understand how to correctly configure kube-dns with flannel on kubernetes 1.10 and containerd as the CRI.</p>

<p>kube-dns fails to run, with the following error:</p>

<pre><code>kubectl -n kube-system logs kube-dns-595fdb6c46-9tvn9 -c kubedns
I0424 14:56:34.944476       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I0424 14:56:35.444469       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
E0424 14:56:35.815863       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:192: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0424 14:56:35.815863       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:189: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
I0424 14:56:35.944444       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I0424 14:56:36.444462       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I0424 14:56:36.944507       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
F0424 14:56:37.444434       1 dns.go:209] Timeout waiting for initialization

kubectl -n kube-system describe pod kube-dns-595fdb6c46-9tvn9
  Type     Reason     Age                 From              Message
  ----     ------     ----                ----              -------
  Warning  Unhealthy  47m (x181 over 3h)  kubelet, worker1  Readiness probe failed: Get http://10.244.0.2:8081/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff    27m (x519 over 3h)  kubelet, worker1  Back-off restarting failed container
  Normal   Killing    17m (x44 over 3h)   kubelet, worker1  Killing container with id containerd://dnsmasq:Container failed liveness probe.. Container will be killed and recreated.
  Warning  Unhealthy  12m (x178 over 3h)  kubelet, worker1  Liveness probe failed: Get http://10.244.0.2:10054/metrics: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff    2m (x855 over 3h)   kubelet, worker1  Back-off restarting failed container
</code></pre>

<p>There is indeed no route to the 10.96.0.1 endpoint:</p>

<pre><code>ip route
default via 10.240.0.254 dev ens160 
10.240.0.0/24 dev ens160  proto kernel  scope link  src 10.240.0.21 
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink 
10.244.0.0/16 dev cni0  proto kernel  scope link  src 10.244.0.1 
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 
10.244.4.0/24 via 10.244.4.0 dev flannel.1 onlink 
10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink
</code></pre>

<p>What is responsible for configuring the cluster service address range and associated routes? Is it the container runtime, the overlay network (flannel in this case), or something else? Where should it be configured?</p>

<p>The <code>10-containerd-net.conflist</code> configures the bridge between the host and my pod network. Can the service network be configured here too?</p>

<pre><code>cat /etc/cni/net.d/10-containerd-net.conflist 
{
  "cniVersion": "0.3.1",
  "name": "containerd-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.244.0.0/16",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
</code></pre>

<p>Edit:</p>

<p>Just came across <a href="https://github.com/kubernetes/kubernetes/issues/27161#issuecomment-225396985" rel="nofollow noreferrer">this</a> from 2016:</p>

<blockquote>
  <p>As of a few weeks ago (I forget the release but it was a 1.2.x where x
  != 0) (#24429) we fixed the routing such that any traffic that arrives
  at a node destined for a service IP will be handled as if it came to a
  node port. This means you should be able to set yo static routes for
  your service cluster IP range to one or more nodes and the nodes will
  act as bridges. This is the same trick most people do with flannel to
  bridge the overlay.</p>
  
  <p>It's imperfect but it works. In the future will will need to get more
  precise with the routing if you want optimal behavior (i.e. not losing
  the client IP), or we will see more non-kube-proxy implementations of
  services.</p>
</blockquote>

<p>Is that still relevant? Do I need to setup a static route for the service CIDR? Or is the issue actually with <code>kube-proxy</code> rather than flannel or containerd?</p>

<p>My flannel configuration:</p>

<pre><code>cat /etc/cni/net.d/10-flannel.conflist 
{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
</code></pre>

<p>And kube-proxy:</p>

<pre><code>[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --cluster-cidr=10.244.0.0/16 \
  --feature-gates=SupportIPVSProxyMode=true \
  --ipvs-min-sync-period=5s \
  --ipvs-sync-period=5s \
  --ipvs-scheduler=rr \
  --kubeconfig=/etc/kubernetes/kube-proxy.conf \
  --logtostderr=true \
  --master=https://192.168.160.1:6443 \
  --proxy-mode=ipvs \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
</code></pre>

<p>Edit:</p>

<p>Having looked at the <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#is-the-kube-proxy-working" rel="nofollow noreferrer">kube-proxy debugging steps</a>, it appears that <code>kube-proxy</code> cannot contact the master. I suspect this is a large part of the problem. I have 3 controller/master nodes behind a HAProxy loadbalancer, which is bound to <code>192.168.160.1:6443</code> and forwards round robin to each of the masters on <code>10.240.0.1[1|2|3]:6443</code>. This can be seen in the output/configs above.</p>

<p>In <code>kube-proxy.service</code>, I have specified <code>--master=192.168.160.1:6443</code>. Why are connections being attempted to port 443? Can I change this - there doesn't seem to be a port flag? Does it need to be port 443 for some reason?</p>

<p>There are two components to this answer, one about running <code>kube-proxy</code> and one about where those :443 URLs are coming from.</p>

<p>First, about <code>kube-proxy</code>: please don't run <code>kube-proxy</code> as a systemd service like that. It is designed to be launched by <code>kubelet</code> <em>in the cluster</em> so that the SDN addresses behave rationally, since they are effectively "fake" addresses. By running <code>kube-proxy</code> outside the control of <code>kubelet</code>, all kinds of weird things are going to happen unless you expend a huge amount of energy to replicate the way that <code>kubelet</code> configures its subordinate docker containers.</p>

<hr>

<p>Now, about that :443 URL:</p>

<blockquote>
  <p><code>E0424 14:56:35.815863       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:192: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host</code></p>
  
  <p>...</p>
  
  <p>Why are connections being attempted to port 443? Can I change this - there doesn't seem to be a port flag? Does it need to be port 443 for some reason?</p>
</blockquote>

<p>That 10.96.0.1 is from the Service CIDR of your cluster, which is (and should be) separate from the Pod CIDR which should be separate from the Node's subnets, etc. The <code>.1</code> of the cluster's Service CIDR is either reserved (or <em>traditionally</em> allocated) to the <code>kubernetes.default.svc.cluster.local</code> <code>Service</code>, with its one <code>Service.port</code> as <code>443</code>.</p>

<p>I'm not super sure why the <code>--master</code> flag doesn't supersede the value in <code>/etc/kubernetes/kube-proxy.conf</code> but since that file is very clearly only supposed to be used by <code>kube-proxy</code>, why not just update the value in the file to remove all doubt?</p>
