<p>I'm running Kubernetes in virtual machines and going through the basic tutorials, currently <a href="https://kubernetes.io/docs/tutorials/stateless-application/guestbook-logs-metrics-with-elk/" rel="noreferrer">Add logging and metrics to the PHP / Redis Guestbook example</a>. I'm trying to install kube-state-metrics:</p>

<pre><code>git clone https://github.com/kubernetes/kube-state-metrics.git kube-state-metrics
kubectl create -f kube-state-metrics/kubernetes
</code></pre>

<p>but it fails. </p>

<pre><code>kubectl describe pod --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7
</code></pre>

<blockquote>
  <p>...</p>
  
  <p>Warning  Unhealthy       28m (x8 over 30m)        kubelet, kubernetes-node1  Readiness probe failed: Get <a href="http://192.168.129.102:8080/healthz" rel="noreferrer">http://192.168.129.102:8080/healthz</a>: dial tcp 192.168.129.102:8080: connect: connection refused</p>
</blockquote>

<pre><code>kubectl logs --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7 -c kube-state-metrics
</code></pre>

<blockquote>
  <p>I0514 17:29:26.980707       1 main.go:85] Using default collectors<br>
  I0514 17:29:26.980774       1 main.go:93] Using all namespace<br>
  I0514 17:29:26.980780       1 main.go:129] metric white-blacklisting: blacklisting the following items:<br>
  W0514 17:29:26.980800       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.<br>
  I0514 17:29:26.983504       1 main.go:169] Testing communication with server<br>
  F0514 17:29:56.984025       1 main.go:137] Failed to create client: ERROR communicating with apiserver: Get <a href="https://10.96.0.1:443/version?timeout=32s" rel="noreferrer">https://10.96.0.1:443/version?timeout=32s</a>: dial tcp 10.96.0.1:443: i/o timeout</p>
</blockquote>

<p>I'm unsure if this 10.96.0.1 IP is correct. My virtual machines are in a bridged network 10.10.10.0/24 and a host-only network 192.168.59.0/24. When initializing Kubernetes I used the argument <code>--pod-network-cidr=192.168.0.0/16</code> so that's one more IP range that I'd expect. But 10.96.0.1 looks unfamiliar.</p>

<p>I'm new to Kubernetes, just doing the basic tutorials, so I don't know what to do now. How to fix it or investigate further?</p>

<hr>

<p>EDIT - additonal info:</p>

<p><code>kubectl get nodes -o wide</code></p>

<pre><code>NAME                STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kubernetes-master   Ready    master   15d   v1.14.1   10.10.10.11   &lt;none&gt;        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
kubernetes-node1    Ready    &lt;none&gt;   15d   v1.14.1   10.10.10.5    &lt;none&gt;        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
kubernetes-node2    Ready    &lt;none&gt;   15d   v1.14.1   10.10.10.98   &lt;none&gt;        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
</code></pre>

<p>The command I used to initialize the cluster:</p>

<pre><code>sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=192.168.0.0/16
</code></pre>

<p>The reason for this is probably overlapping of Pod network with Node network - you set Pod network CIDR to 192.168.0.0/16 which your host-only network will be included into as its address is 192.168.59.0/24. </p>

<p>To solve this you can either change the pod network CIDR to 192.168.0.0/24 (<strong>it is not recommended as this will give you only 255 addresses for your pod networking</strong>)</p>

<p>You can also use different range for your Calico. If you want to do it on a running cluster here is an <a href="https://docs.projectcalico.org/v3.6/networking/changing-ip-pools" rel="nofollow noreferrer">instruction</a>. </p>

<p>Also other way I tried:</p>

<p>edit Calico <a href="https://docs.projectcalico.org/v3.7/manifests/calico.yaml" rel="nofollow noreferrer">manifest</a> to different range (for example 10.0.0.0/8)  - <code>sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=10.0.0.0/8)</code> and apply it after the init. </p>

<p>Another way would be using different CNI like Flannel (which uses 10.244.0.0/16).  </p>

<p>You can find more information about ranges of CNI plugins <a href="https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network" rel="nofollow noreferrer">here</a>. </p>
