<p>I am trying to set up monitoring stack (prometheus + alertmanager + node_exporter etc) via <code>helm install stable/prometheus</code> onto a raspberry pi k8s cluster (1 master + 3 worker nodes) which i set up. </p>

<p>Managed to get all the required pods running.</p>

<pre><code>pi-monitoring-prometheus-alertmanager-767cd8bc65-89hxt   2/2     Running            0          131m    10.17.2.56      kube2   &lt;none&gt;           &lt;none&gt;
pi-monitoring-prometheus-node-exporter-h86gt             1/1     Running            0          131m    192.168.1.212   kube2   &lt;none&gt;           &lt;none&gt;
pi-monitoring-prometheus-node-exporter-kg957             1/1     Running            0          131m    192.168.1.211   kube1   &lt;none&gt;           &lt;none&gt;
pi-monitoring-prometheus-node-exporter-x9wgb             1/1     Running            0          131m    192.168.1.213   kube3   &lt;none&gt;           &lt;none&gt;
pi-monitoring-prometheus-pushgateway-799d4ff9d6-rdpkf    1/1     Running            0          131m    10.17.3.36      kube1   &lt;none&gt;           &lt;none&gt;
pi-monitoring-prometheus-server-5d989754b6-gp69j         2/2     Running            0          98m     10.17.1.60      kube3   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>however after port-forwarding prometheus server port 9090 and navigating to <code>Targets</code> page, i realized none of the node_exporters are registered.</p>

<p>Digging through the logs, i found this</p>

<pre><code>evel=error ts=2020-04-12T05:15:05.083Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:333: Failed to list *v1.Node: Get https://10.18.0.1:443/api/v1/nodes?limit=500&amp;resourceVersion=0: dial tcp 10.18.0.1:443: i/o timeout"
level=error ts=2020-04-12T05:15:05.084Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:299: Failed to list *v1.Service: Get https://10.18.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0: dial tcp 10.18.0.1:443: i/o timeout"
level=error ts=2020-04-12T05:15:05.084Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:261: Failed to list *v1.Endpoints: Get https://10.18.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0: dial tcp 10.18.0.1:443: i/o timeout"
level=error ts=2020-04-12T05:15:05.085Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:262: Failed to list *v1.Service: Get https://10.18.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0: dial tcp 10.18.0.1:443: i/o timeout"
</code></pre>

<p><strong>Question: why is the prometheus pod unable to call the apiserver endpoints?  Not really sure where was the configuration done wrongly</strong></p>

<p>Followed through <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/" rel="nofollow noreferrer">debug guide</a> and realized individual nodes are unable to resolve services on other nodes. </p>

<p>Been troubleshooting for the past 1 day reading various sources but to be honest, i am not even sure where to begin with.</p>

<p>These are the pods running in <code>kube-system</code> namespace. Hope this will give a better idea of how my system is set up.</p>

<pre><code>pi@kube4:~ $ kubectl get pods -n kube-system -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
coredns-66bff467f8-nzvq8        1/1     Running   0          13d   10.17.0.2       kube4   &lt;none&gt;           &lt;none&gt;
coredns-66bff467f8-z7wdb        1/1     Running   0          13d   10.17.0.3       kube4   &lt;none&gt;           &lt;none&gt;
etcd-kube4                      1/1     Running   0          13d   192.168.1.214   kube4   &lt;none&gt;           &lt;none&gt;
kube-apiserver-kube4            1/1     Running   2          13d   192.168.1.214   kube4   &lt;none&gt;           &lt;none&gt;
kube-controller-manager-kube4   1/1     Running   2          13d   192.168.1.214   kube4   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-arm-8g9fb       1/1     Running   1          13d   192.168.1.212   kube2   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-arm-c5qt9       1/1     Running   0          13d   192.168.1.214   kube4   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-arm-q5pln       1/1     Running   1          13d   192.168.1.211   kube1   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-arm-tkmn6       1/1     Running   1          13d   192.168.1.213   kube3   &lt;none&gt;           &lt;none&gt;
kube-proxy-4zjjh                1/1     Running   0          13d   192.168.1.213   kube3   &lt;none&gt;           &lt;none&gt;
kube-proxy-6mk2z                1/1     Running   0          13d   192.168.1.211   kube1   &lt;none&gt;           &lt;none&gt;
kube-proxy-bbr8v                1/1     Running   0          13d   192.168.1.212   kube2   &lt;none&gt;           &lt;none&gt;
kube-proxy-wfsbm                1/1     Running   0          13d   192.168.1.214   kube4   &lt;none&gt;           &lt;none&gt;
kube-scheduler-kube4            1/1     Running   3          13d   192.168.1.214   kube4   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p><a href="https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md" rel="nofollow noreferrer">Flannel documentation</a> states:</p>

<blockquote>
  <p>NOTE: If <code>kubeadm</code> is used, then pass <code>--pod-network-cidr=10.244.0.0/16</code> to <code>kubeadm init</code> to ensure that the <code>podCIDR</code> is set.</p>
</blockquote>

<p>This is because flannel ConfigMap by default is configured to work on <code>"Network": "10.244.0.0/16"</code></p>

<p>You have configured your kubeadm with <code>--pod-network-cidr=10.17.0.0/16</code> now this needs to be configured in flannel ConfigMap <code>kube-flannel-cfg</code> to look like this:</p>

<pre class="lang-yaml prettyprint-override"><code>kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.17.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
</code></pre>

<p>Thanks to <a href="https://stackoverflow.com/users/5689112/kitt">@kitt</a> for his debugging help.</p>
