<p>I have a Kubernetes cluster set up using kubeadm.
I installed prometheus and node-exporter on top of it based on:</p>
<ul>
<li><a href="https://github.com/bibinwilson/kubernetes-prometheus" rel="nofollow noreferrer">https://github.com/bibinwilson/kubernetes-prometheus</a></li>
<li><a href="https://github.com/bibinwilson/kubernetes-node-exporter" rel="nofollow noreferrer">https://github.com/bibinwilson/kubernetes-node-exporter</a></li>
</ul>
<p>The pods seem to be running properly:</p>
<pre><code> kubectl get pods --namespace=monitoring -o wide
NAME                                     READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
node-exporter-jk2sd                      1/1     Running   0          90m   192.168.5.20   work03   &lt;none&gt;           &lt;none&gt;
node-exporter-jldrx                      1/1     Running   0          90m   192.168.5.17   work04   &lt;none&gt;           &lt;none&gt;
node-exporter-mgtld                      1/1     Running   0          90m   192.168.5.15   work01   &lt;none&gt;           &lt;none&gt;
node-exporter-tq7bx                      1/1     Running   0          90m   192.168.5.41   work02   &lt;none&gt;           &lt;none&gt;
prometheus-deployment-5d79b5f65b-tkpd2   1/1     Running   0          91m   192.168.5.40   work02   &lt;none&gt;           &lt;none&gt;

</code></pre>
<p>I can see the endpoints, as well:</p>
<pre><code>kubectl get endpoints -n monitoring
NAME            ENDPOINTS                                                           AGE
node-exporter   192.168.5.15:9100,192.168.5.17:9100,192.168.5.20:9100 + 1 more...   5m3s
</code></pre>
<p>I also did: <code>kubectl port-forward prometheus-deployment-5d79b5f65b-tkpd2 8080:9090 -n monitoring</code> and when I access the prometheus web UI &gt; Status &gt; Targets, I don't find node-exporters there. When I start typing a query for a metric reported by node-exporter, it doesn't automatically show up in the query editor.</p>
<p>Logs coming from the prometheus pod seem to have a lot of errors:</p>
<pre><code>kubectl logs prometheus-deployment-5d79b5f65b-tkpd2 -n monitoring
level=info ts=2021-08-11T16:24:21.743Z caller=main.go:428 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.29.1, branch=HEAD, revision=dcb07e8eac34b5ea37cd229545000b857f1c1637)&quot;
level=info ts=2021-08-11T16:24:21.743Z caller=main.go:433 build_context=&quot;(go=go1.16.7, user=root@364730518a4e, date=20210811-14:48:27)&quot;
level=info ts=2021-08-11T16:24:21.743Z caller=main.go:434 host_details=&quot;(Linux 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 prometheus-deployment-5d79b5f65b-tkpd2 (none))&quot;
level=info ts=2021-08-11T16:24:21.743Z caller=main.go:435 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
level=info ts=2021-08-11T16:24:21.743Z caller=main.go:436 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
level=info ts=2021-08-11T16:24:21.745Z caller=web.go:541 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
level=info ts=2021-08-11T16:24:21.745Z caller=main.go:812 msg=&quot;Starting TSDB ...&quot;
level=info ts=2021-08-11T16:24:21.748Z caller=tls_config.go:191 component=web msg=&quot;TLS is disabled.&quot; http2=false
level=info ts=2021-08-11T16:24:21.753Z caller=head.go:815 component=tsdb msg=&quot;Replaying on-disk memory mappable chunks if any&quot;
level=info ts=2021-08-11T16:24:21.753Z caller=head.go:829 component=tsdb msg=&quot;On-disk memory mappable chunks replay completed&quot; duration=4.15µs
level=info ts=2021-08-11T16:24:21.753Z caller=head.go:835 component=tsdb msg=&quot;Replaying WAL, this may take a while&quot;
level=info ts=2021-08-11T16:24:21.754Z caller=head.go:892 component=tsdb msg=&quot;WAL segment loaded&quot; segment=0 maxSegment=0
level=info ts=2021-08-11T16:24:21.754Z caller=head.go:898 component=tsdb msg=&quot;WAL replay completed&quot; checkpoint_replay_duration=75.316µs wal_replay_duration=451.769µs total_replay_duration=566.051µs
level=info ts=2021-08-11T16:24:21.756Z caller=main.go:839 fs_type=EXT4_SUPER_MAGIC
level=info ts=2021-08-11T16:24:21.756Z caller=main.go:842 msg=&quot;TSDB started&quot;
level=info ts=2021-08-11T16:24:21.756Z caller=main.go:969 msg=&quot;Loading configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2021-08-11T16:24:21.757Z caller=kubernetes.go:282 component=&quot;discovery manager scrape&quot; discovery=kubernetes msg=&quot;Using pod service account via in-cluster config&quot;
level=info ts=2021-08-11T16:24:21.759Z caller=kubernetes.go:282 component=&quot;discovery manager scrape&quot; discovery=kubernetes msg=&quot;Using pod service account via in-cluster config&quot;
level=info ts=2021-08-11T16:24:21.762Z caller=kubernetes.go:282 component=&quot;discovery manager scrape&quot; discovery=kubernetes msg=&quot;Using pod service account via in-cluster config&quot;
level=info ts=2021-08-11T16:24:21.764Z caller=main.go:1006 msg=&quot;Completed loading of configuration file&quot; filename=/etc/prometheus/prometheus.yml totalDuration=7.940972ms db_storage=607ns remote_storage=1.251µs web_handler=283ns query_engine=694ns scrape=227.668µs scrape_sd=6.081132ms notify=27.11µs notify_sd=16.477µs rules=648.58µs
level=info ts=2021-08-11T16:24:21.764Z caller=main.go:784 msg=&quot;Server is ready to receive web requests.&quot;
level=error ts=2021-08-11T16:24:51.765Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:24:51.765Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:24:51.765Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get \&quot;https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:24:51.766Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:24:51.766Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get \&quot;https://10.96.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:22.587Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get \&quot;https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:22.855Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:23.153Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:23.261Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:23.335Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get \&quot;https://10.96.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:54.814Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:55.282Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get \&quot;https://10.96.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:55.516Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get \&quot;https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:55.934Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:25:56.442Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:26:30.058Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:26:30.204Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:26:30.246Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get \&quot;https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:26:30.879Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:26:31.479Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get \&quot;https://10.96.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:27:09.673Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:27:09.835Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get \&quot;https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:27:10.467Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:27:11.170Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:27:12.684Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get \&quot;https://10.96.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:27:55.324Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: Get \&quot;https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:28:01.550Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:28:01.621Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:28:04.801Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:28:05.598Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get \&quot;https://10.96.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:28:57.256Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
level=error ts=2021-08-11T16:29:04.688Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.3/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://10.96.0.1:443/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: dial tcp 10.96.0.1:443: i/o timeout&quot;
</code></pre>
<p>Is there a way to solve this issue and make node-exporters show up in the targets?</p>
<p>Version details:</p>
<pre><code>kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.5&quot;, GitCommit:&quot;6b1d87acf3c8253c123756b9e61dac642678305f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-18T01:10:43Z&quot;, GoVersion:&quot;go1.15.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.9&quot;, GitCommit:&quot;7a576bc3935a6b555e33346fd73ad77c925e9e4a&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-07-15T20:56:38Z&quot;, GoVersion:&quot;go1.15.14&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>Edit:
The cluster was set up as follows:</p>
<pre><code>sudo kubeadm reset
sudo rm $HOME/.kube/config
sudo kubeadm init --pod-network-cidr=192.168.5.0/24
mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
</code></pre>
<p>It is using flannel.</p>
<p>flannel pods are running:</p>
<pre><code>kube-flannel-ds-45qwf                1/1     Running   0          31h   x.x.x.41   work01   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-4rwzj                1/1     Running   0          31h   x.x.x.40   mast01   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-8fdtt                1/1     Running   24         31h   x.x.x.43   work03   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-8hl5f                1/1     Running   23         31h   x.x.x.44   work04   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-xqtrd                1/1     Running   0          31h   x.x.x.42   work02   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>The issue is related to SDN not working properly.</p>
<p>As a general rule, troubleshooting this, we would check the SDN pods (calico, weave, or in this case flannel), are they healthy, any errors in their logs, ...</p>
<p>Check iptables (<code>iptables -nL</code>) and ipvs (<code>ipvsadm -l n</code>) configuration nodes.</p>
<p>Restart SDN pods, as well as kube-proxy, if you still didn't find anything.</p>
<p>Now, on this specific case, we're not suffering from an outage: cluster is freshly deployed, it's likely the SDN never worked at all - though this may not be obvious, with a kubeadm deployment, that doesn't ship with other pods than the defaults, most of which using host networking.</p>
<p>The kubeadm init command mentions that pod CIDR is some 192.168.5.0/24, which brings two remarks:</p>
<ul>
<li><p>with all SDN: the pod CIDR is a subnet that will be split into smaller subnets (usually /24 or /25). Each range being statically allocated to Nodes when they first join your cluster</p>
</li>
<li><p>running flannel SDN: kubeadm init should include a <code>--pod-network-cidr</code> argument that MUST match the subnet configured in the <code>kube-flannel-cfg</code> ConfigMap, see <code>net-conf.json</code> key.</p>
</li>
</ul>
<p>Though I'm unfamiliar with the process of fixing this, there seem to be an answer on ServerFault that gives some instructions, which sounds right: <a href="https://serverfault.com/a/977401/293779">https://serverfault.com/a/977401/293779</a></p>
