<p>I've updated my cluster to v1.18.1. Some applications have API access but they returned an error that the API is not reachable. A similar error, is returned by the ping command. Here are two outputs, the first from a go application and the second from a ping command against the API.</p>
<pre class="lang-sh prettyprint-override"><code>I0623 15:58:57.317985      23 trace.go:201] Trace[163617342]: &quot;Reflector ListAndWatch&quot; name:pkg/mod/k8s.io/client-go@v0.18.1/tools/cache/reflector.go:125 (23-Jun-2020 15:58:00.317) (total time: 30000ms):
Trace[163617342]: [30.000517214s] [30.000517214s] END
E0623 15:58:57.318003      23 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.1/tools/cache/reflector.go:125: Failed to list *v1.Service: Get &quot;https://10.96.0.1:443/api/v1/namespaces/default/services?limit=500&amp;resourceVersion=0&quot;: dial tcp 10.96.0.1:443: i/o timeout
</code></pre>
<pre class="lang-sh prettyprint-override"><code>$ kubectl exec -it network-tools -- ping 10.96.0.1
ping: socket: Operation not permitted
command terminated with exit code 2
</code></pre>
<p>I can rule out that the API is basically not accessible. I can access it via kubectl.</p>
<p>As network plugin I use flannel. To be on the safe side, I have re-played the official flannel YAML to make sure that there is no possible update. But this did not help.</p>
<p>Now I just don't know where the mistake comes from. To give supporters some more info about the cluster here are some details.</p>
<p>Volker</p>
<h1>Nodes</h1>
<pre class="lang-sh prettyprint-override"><code>$ kgno -o wide
NAME         STATUS   ROLES    AGE    VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
orbisos001   Ready    master   4h5m   v1.18.1   192.168.179.100   &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-1127.el7.x86_64        cri-o://1.18.1
orbisos002   Ready    &lt;none&gt;   4h3m   v1.18.1   192.168.179.111   &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-1127.10.1.el7.x86_64   cri-o://1.18.1
</code></pre>
<h1>Services</h1>
<pre class="lang-sh prettyprint-override"><code>$ kgsv --all-namespaces
NAMESPACE     NAME         TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
default       kubernetes   ClusterIP      10.96.0.1      &lt;none&gt;            443/TCP                      4h10m
default       proxy        LoadBalancer   10.107.54.36   192.168.179.101   80:31414/TCP,443:32154/TCP   3h15m
kube-system   kube-dns     ClusterIP      10.96.0.10     &lt;none&gt;            53/UDP,53/TCP,9153/TCP       4h10m
</code></pre>
<h1>/etc/cni/net.d/100-crio-bridge.conf</h1>
<pre><code>{
    &quot;cniVersion&quot;: &quot;0.3.1&quot;,
    &quot;name&quot;: &quot;crio&quot;,
    &quot;type&quot;: &quot;bridge&quot;,
    &quot;bridge&quot;: &quot;cni0&quot;,
    &quot;isGateway&quot;: true,
    &quot;ipMasq&quot;: true,
    &quot;hairpinMode&quot;: true,
    &quot;ipam&quot;: {
        &quot;type&quot;: &quot;host-local&quot;,
        &quot;routes&quot;: [
            { &quot;dst&quot;: &quot;0.0.0.0/0&quot; },
            { &quot;dst&quot;: &quot;1100:200::1/24&quot; }
        ],
        &quot;ranges&quot;: [
            [{ &quot;subnet&quot;: &quot;10.85.0.0/16&quot; }],
            [{ &quot;subnet&quot;: &quot;1100:200::/24&quot; }]
        ]
    }
}
</code></pre>
<h1>RPM-Versions</h1>
<pre class="lang-sh prettyprint-override"><code>$ yum list installed | grep -e kube -e cri-
cri-o.x86_64                       2:1.18.1-1.1.el7                 @crio
cri-tools.x86_64                   1.13.0-1.rhaos4.1.gitc06001f.el7 @tools
kubeadm.x86_64                     1.18.1-0                         @kubernetes
kubectl.x86_64                     1.18.1-0                         @kubernetes
kubelet.x86_64                     1.18.1-0                         @kubernetes
</code></pre>
<h1>KUBELET_EXTRA_ARGS</h1>
<pre class="lang-sh prettyprint-override"><code>$ cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice
</code></pre>

Volker Raschek
<p>After a long search I have found a solution. The container kube-flannel-amd64 in the kube-system namespace initially throws the error that it has no permission to access iptables. So the IP packets are not routed over the VXLAN. This caused into the timeout error.</p>
<p>To give the container access to iptables of the hostsystem I changed the official kube-flannel.yml from <code>privileged: false</code> to <code>true</code>.</p>
<pre class="lang-yaml prettyprint-override"><code>securityContext:
  privileged: true
  capabilities:
    add: [&quot;NET_ADMIN&quot;]
</code></pre>
<p>After a new deploy of the YAML file the rules are successfully created:</p>
<pre class="lang-sh prettyprint-override"><code>I0625 08:53:13.166567       1 vxlan_network.go:60] watching for new subnet leases
I0625 08:53:13.168489       1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0625 08:53:13.168501       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0625 08:53:13.169149       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0625 08:53:13.170435       1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN
I0625 08:53:13.171086       1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
I0625 08:53:13.262424       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0625 08:53:13.263101       1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0625 08:53:13.263109       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0625 08:53:13.264018       1 iptables.go:167] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0625 08:53:13.264195       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0625 08:53:13.264883       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0625 08:53:13.267062       1 iptables.go:155] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0625 08:53:13.267781       1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN
I0625 08:53:13.363094       1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
</code></pre>
<p>My application can now connect the kubernetes API.</p>
