title: Kubernetes: Unable to access flannel pod
body: <p>I am new to Kubernetes.  I have set up 3 Ubuntu 20.04.2 LTS VMs on Oracle Virtualbox Manager.</p>
  <p>I have installed docker, kubelet, kubeadm, and kubectl in all 3 VMs according to the following documentation.<br />
  <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/" rel="nofollow noreferrer">https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/</a></p>
  <p>And I created cluster using the following link:
  <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/" rel="nofollow noreferrer">https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/</a></p>
  <p>I used the following commands to setup flannel</p>
  <pre><code>$ wget https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
  $ kubectl create -f kube-flannel.yml
  </code></pre>
  <p>Everything looks fine.</p>
  <pre><code>root@master-node:~/k8s# kubectl get nodes -o wide
  NAME          STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
  master-node   Ready    control-plane,master   23h   v1.20.5   192.168.108.10   &lt;none&gt;        Ubuntu 20.04.2 LTS   5.4.0-70-generic   docker://19.3.15
  node-1        Ready    &lt;none&gt;                 10h   v1.20.5   192.168.108.11   &lt;none&gt;        Ubuntu 20.04.2 LTS   5.4.0-70-generic   docker://19.3.15
  node-2        Ready    &lt;none&gt;                 10h   v1.20.5   192.168.108.12   &lt;none&gt;        Ubuntu 20.04.2 LTS   5.4.0-70-generic   docker://19.3.15
  
  </code></pre>
  <p>I then create nginx deployment with 3 replicas.</p>
  <pre><code>root@master-node:~/k8s# kubectl get po -o wide
  NAME                            READY   STATUS    RESTARTS   AGE    IP           NODE     NOMINATED NODE   READINESS GATES
  dnsutils                        1/1     Running   2          127m   10.244.2.8   node-2   &lt;none&gt;           &lt;none&gt;
  nginx-deploy-7848d4b86f-4nvg7   1/1     Running   0          9m8s   10.244.2.9   node-2   &lt;none&gt;           &lt;none&gt;
  nginx-deploy-7848d4b86f-prj7g   1/1     Running   0          9m8s   10.244.1.9   node-1   &lt;none&gt;           &lt;none&gt;
  nginx-deploy-7848d4b86f-r95hq   1/1     Running   0          9m8s   10.244.1.8   node-1   &lt;none&gt;           &lt;none&gt;
  </code></pre>
  <p>The problem shows only when I tried to curl the nginx pods.  It is not responsive.</p>
  <pre><code>root@master-node:~/k8s# curl 10.244.2.9
  ^C
  </code></pre>
  <p>I then login to the pod and confirmed that nginx is up.</p>
  <pre><code>root@master-node:~/k8s# kubectl exec -it nginx-deploy-7848d4b86f-4nvg7  -- /bin/bash
  root@nginx-deploy-7848d4b86f-4nvg7:/# curl 127.0.0.1
  &lt;!DOCTYPE html&gt;
  &lt;html&gt;
  &lt;head&gt;
  &lt;title&gt;Welcome to nginx!&lt;/title&gt;
  &lt;style&gt;
      body {
          width: 35em;
          margin: 0 auto;
          font-family: Tahoma, Verdana, Arial, sans-serif;
      }
  &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
  &lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
  &lt;p&gt;If you see this page, the nginx web server is successfully installed and
  working. Further configuration is required.&lt;/p&gt;
  
  &lt;p&gt;For online documentation and support please refer to
  &lt;a href=&quot;http://nginx.org/&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
  Commercial support is available at
  &lt;a href=&quot;http://nginx.com/&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;
  
  &lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  root@nginx-deploy-7848d4b86f-4nvg7:/# exit
  exit
  
  </code></pre>
  <p>Here is the result of kubectl describe pod on one of the pods:</p>
  <pre><code>root@master-node:~/k8s# kubectl describe pod nginx-deploy-7848d4b86f-4nvg7
  Name:         nginx-deploy-7848d4b86f-4nvg7
  Namespace:    default
  Priority:     0
  Node:         node-2/192.168.108.12
  Start Time:   Sun, 28 Mar 2021 04:49:15 +0000
  Labels:       app=nginx
                pod-template-hash=7848d4b86f
  Annotations:  &lt;none&gt;
  Status:       Running
  IP:           10.244.2.9
  IPs:
    IP:           10.244.2.9
  Controlled By:  ReplicaSet/nginx-deploy-7848d4b86f
  Containers:
    nginx:
      Container ID:   docker://f6322e65cb98e54cc220a786ffb7c967bbc07d80fe8d118a19891678109680d8
      Image:          nginx
      Image ID:       docker-pullable://nginx@sha256:b0ea179ab61c789ce759dbe491cc534e293428ad232d00df83ce44bf86261179
      Port:           80/TCP
      Host Port:      0/TCP
      State:          Running
        Started:      Sun, 28 Mar 2021 04:49:19 +0000
      Ready:          True
      Restart Count:  0
      Environment:    &lt;none&gt;
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-xhkzx (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             True
    ContainersReady   True
    PodScheduled      True
  Volumes:
    default-token-xhkzx:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-xhkzx
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  &lt;none&gt;
  Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type    Reason     Age   From               Message
    ----    ------     ----  ----               -------
    Normal  Scheduled  25m   default-scheduler  Successfully assigned default/nginx-deploy-7848d4b86f-4nvg7 to node-2
    Normal  Pulling    25m   kubelet            Pulling image &quot;nginx&quot;
    Normal  Pulled     25m   kubelet            Successfully pulled image &quot;nginx&quot; in 1.888247052s
    Normal  Created    25m   kubelet            Created container nginx
    Normal  Started    25m   kubelet            Started container nginx
  </code></pre>
  <p>I tried to troubleshoot by using: <a href="https://www.praqma.com/stories/debugging-kubernetes-networking/" rel="nofollow noreferrer">Debugging Kubernetes Networking</a></p>
  <pre><code>root@master-node:~/k8s# ip link list
  1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
      link/ether 08:00:27:db:6f:21 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
      link/ether 08:00:27:90:88:7c brd ff:ff:ff:ff:ff:ff
  4: docker0: &lt;NO-CARRIER,BROADCAST,MULTICAST,UP&gt; mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
      link/ether 02:42:1d:21:66:20 brd ff:ff:ff:ff:ff:ff
  5: kube-ipvs0: &lt;BROADCAST,NOARP&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default
      link/ether 4a:df:fb:be:7b:0e brd ff:ff:ff:ff:ff:ff
  6: flannel.1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
      link/ether 02:48:db:46:53:60 brd ff:ff:ff:ff:ff:ff
  7: cni0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
      link/ether fa:29:13:98:2c:31 brd ff:ff:ff:ff:ff:ff
  8: vethc2e0fa86@if3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
      link/ether 7a:66:b0:97:db:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0
  9: veth3eb514e1@if3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
      link/ether 3e:3c:9d:20:5c:42 brd ff:ff:ff:ff:ff:ff link-netnsid 1
  11: veth0@if10: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
      link/ether 02:35:f0:fb:e3:b1 brd ff:ff:ff:ff:ff:ff link-netns test1
  root@master-node:~/k8s# kubectl create -f nwtool-deployment.yaml
  deployment.apps/nwtool-deploy created
  root@master-node:~/k8s# kubectl get po
  NAME                             READY   STATUS    RESTARTS   AGE
  nwtool-deploy-6d8c99644b-fq6gv   1/1     Running   0          14s
  nwtool-deploy-6d8c99644b-fwc6d   1/1     Running   0          14s
  root@master-node:~/k8s# ^C
  root@master-node:~/k8s# kubectl exec -it nwtool-deploy-6d8c99644b-fq6gv -- ip link list
  1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  3: eth0@if13: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue state UP mode DEFAULT group default
      link/ether 2e:02:b6:97:2f:10 brd ff:ff:ff:ff:ff:ff
  root@master-node:~/k8s# kubectl exec -it nwtool-deploy-6d8c99644b-fwc6d -- ip link list
  1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  3: eth0@if14: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue state UP mode DEFAULT group default
      link/ether 82:21:fa:aa:34:27 brd ff:ff:ff:ff:ff:ff
  root@master-node:~/k8s# ip link list
  1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
      link/ether 08:00:27:db:6f:21 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
      link/ether 08:00:27:90:88:7c brd ff:ff:ff:ff:ff:ff
  4: docker0: &lt;NO-CARRIER,BROADCAST,MULTICAST,UP&gt; mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
      link/ether 02:42:1d:21:66:20 brd ff:ff:ff:ff:ff:ff
  5: kube-ipvs0: &lt;BROADCAST,NOARP&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default
      link/ether 4a:df:fb:be:7b:0e brd ff:ff:ff:ff:ff:ff
  6: flannel.1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
      link/ether 02:48:db:46:53:60 brd ff:ff:ff:ff:ff:ff
  7: cni0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
      link/ether fa:29:13:98:2c:31 brd ff:ff:ff:ff:ff:ff
  8: vethc2e0fa86@if3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
      link/ether 7a:66:b0:97:db:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0
  9: veth3eb514e1@if3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
      link/ether 3e:3c:9d:20:5c:42 brd ff:ff:ff:ff:ff:ff link-netnsid 1
  11: veth0@if10: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
      link/ether 02:35:f0:fb:e3:b1 brd ff:ff:ff:ff:ff:ff link-netns test1
  root@master-node:~/k8s#
  
  </code></pre>
  <p>It looks that no veth pairs were created for the new pod on the master node.  Any idea how to resolve this?  Any help will be greatly appreciated.  Thank you!</p>