<p>Installed a kubernetes cluster with Calico, CoreDNS.</p>
<p>Check one CoreDNS's event message got</p>
<pre><code>Readiness probe failed: HTTP probe failed with statuscode: 503
</code></pre>
<p>Under <code>/var/lib/cni/networks/</code> directory there is nothing. Why? How to fix?</p>
<p>Even all the pods' status is Running, but worrying about its healthy.</p>
<p>logs</p>
<pre><code># kubectl logs coredns-1308140hfw -n kube-system
[INFO] plugin/ready: Still waiting on: &quot;kubernetes&quot;
.:53
[INFO] plugin/reload: Running configuration MD5 = 20328084ha6966e76816bcd928foa
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] plugin/ready: Still waiting on: &quot;kubernetes&quot;
[INFO] plugin/ready: Still waiting on: &quot;kubernetes&quot;
I0804 08:18:03.874045       1 trace.go:116] Trace[336122540]: &quot;Reflector ListAndWatch&quot; name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2020-08-04 08:17:33.872753993 +0000 UTC m=+0.038838328) (total time: 30.001059939s):
Trace[336122540]: [30.001059939s] [30.001059939s] END
E0804 08:18:03.874108       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get &quot;https://[IPv6]:443/api/v1/endpoints?limit=500&amp;resourceVersion=0&quot;: dial tcp [IPv6]:443: i/o timeout
I0804 08:18:03.874047       1 trace.go:116] Trace[208240456]: &quot;Reflector ListAndWatch&quot; name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2020-08-04 08:17:33.872755558 +0000 UTC m=+0.038839930) (total time: 30.001213767s):
Trace[208240456]: [30.001213767s] [30.001213767s] END
E0804 08:18:03.874137       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get &quot;https://[IPv6]:443/api/v1/namespaces?limit=500&amp;resourceVersion=0&quot;: dial tcp [IPv6]:443: i/o timeout
I0804 08:18:03.874214       1 trace.go:116] Trace[1106410694]: &quot;Reflector ListAndWatch&quot; name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2020-08-04 08:17:33.872753715 +0000 UTC m=+0.038838086) (total time: 30.001438405s):
Trace[1106410694]: [30.001438405s] [30.001438405s] END
E0804 08:18:03.874248       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Get &quot;https://[IPv6]:443/api/v1/services?limit=500&amp;resourceVersion=0&quot;: dial tcp [IPv6]:443: i/o timeout
</code></pre>

rawmain
<p>How readiness probe works?</p>
<blockquote>
<p>Sometimes, applications are temporarily unable to serve traffic. For
example, an application might need to load large data or configuration
files during startup. In such cases, you don't want to kill the
application, but you don’t want to send it requests either. Kubernetes
provides readiness probes to detect and mitigate these situations. A
pod with containers reporting that they are not ready does not receive
traffic through Kubernetes Services.</p>
</blockquote>
<p>When a pod is started, Kubernetes can be configured to wait for a configurable amount of time to pass before performing the first readiness check. After that, it invokes the probe periodically and acts based on the result of the readiness probe. If a pod reports that it’s not ready, it’s removed from the service. If the pod then becomes ready again, it’s re-added.</p>
<p>This means that in your situation your <code>coreDNS</code> pod was not entirely running but Kubernetes already started sending probes to check it's <code>readiness</code>.</p>
<p>Thanks to those probes, when you have a couple of replicas, Kubernetes will direct traffic only to those which are healthy (with a successful probe).</p>
<p>PS. My <code>/var/lib/cni/networks/</code>  directory is also empty.</p>
