<p>We are trying to create POD but the Pod's status struck at ContainerCreating for long time.</p>
<p>This is the output we got after running the command: kubectl describe pod</p>
<pre><code>Name:           demo-6c59fb8f77-9x6sr
Namespace:      default
Priority:       0
Node:           k8-slave2/10.0.0.5
Start Time:     Wed, 23 Dec 2020 10:16:23 +0000
Labels:         app=demo
                pod-template-hash=6c59fb8f77
Annotations:    &lt;none&gt;
Status:         Pending
IP:
IPs:            &lt;none&gt;
Controlled By:  ReplicaSet/demo-6c59fb8f77
Containers:
  private-docker-registry:
    Container ID:
    Image:          private-docker-registry:5000/mahin/mof-docker-demo:v1
    Image ID:
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p94zw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-p94zw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p94zw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  

  Events:
      Type     Reason                  Age                  From               Message
      ----     ------                  ----                 ----               -------
      Normal   Scheduled               10m                  default-scheduler  Successfully assigned default/demo-6c59fb8f77-9x6sr to k8-slave2
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;95e72bfc6f6c13de7f5c96eb76b012c2e6639ca03f4c2f270b23ed1a09b90413&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;566370012e4a1d32af2ef9035ff64d743cd81f36f25d2724e7b033e393b8247e&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;7d499e40f572cfc29ecfb44f8376493df56a44213b1c1e9333b65499a0c288cd&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;53241e64de1e4470712b4061e2c82f44916d654bc532f8f1d12e5d5d4e136914&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;fd168faab4546f988dc38fc56df2f71cf80c922e86d3f869be15a43f08328f99&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;e578afe329abb0cba64802dfa480e00f2bbbb8c80be537791c24a31c853eb62f&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;a3cb32dba55907ca907fc4f38f7ca05ef6db10a6af2dd1fa3c4db166e4ab9ffe&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;7e4368ba8ec460b3c94de24ab0a04b6c799eb28df885cbbacfc3bb3ffa8c1e67&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m (x4 over 10m)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;c4aaa8f8cd2dc1eff788baf04774c4ecc845568d00ed1b386df311ec224eb6f3&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
      Normal   SandboxChanged          56s (x551 over 10m)  kubelet            Pod sandbox changed, it will be killed and re-created.



azureuser@k8-master:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                demo-6c59fb8f77-2jq6k                        0/1     ContainerCreating   0          5m23s
kube-system            coredns-f9fd979d6-q8s9b                      1/1     Running             2          27h
kube-system            coredns-f9fd979d6-qnm4j                      1/1     Running             2          27h
kube-system            etcd-k8-master                               1/1     Running             2          27h
kube-system            kube-apiserver-k8-master                     1/1     Running             3          27h
kube-system            kube-controller-manager-k8-master            1/1     Running             3          27h
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
kube-system            kube-proxy-4mb47                             1/1     Running             2          27h
kube-system            kube-proxy-54m9b                             1/1     Running             2          27h
kube-system            kube-proxy-wdxfz                             1/1     Running             1          27h
kube-system            kube-scheduler-k8-master                     1/1     Running             3          27h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-zmlvs   0/1     ContainerCreating   0          27h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-cnsvn         0/1     ContainerCreating   0          6h3m
</code></pre>
<p>To fix the flannel crashloopbackoff we did Kubeadm reset and after some time this problem showed up again.</p>
<p>Current we are working with one master and two worker node.</p>
<p>My cluster details as follows:</p>
<pre><code>azureuser@k8-master:~$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://52.150.11.168:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
</code></pre>
<p>Docker version:</p>
<pre><code>azureuser@k8-master:~$ sudo docker version
[sudo] password for azureuser: 
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Wed Oct 14 19:00:27 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Wed Oct 14 16:52:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:
</code></pre>
<p>kubeadm version :</p>
<pre><code>azureuser@k8-master:~$ kubeadm version
kubeadm version: &amp;version.Info{Major:&quot;1&quot;, Minor:&quot;19&quot;, GitVersion:&quot;v1.19.4&quot;, GitCommit:&quot;d360454c9bcd1634cf4cc52d1867af5491dc9c5f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2020-11-11T13:15:05Z&quot;, GoVersion:&quot;go1.15.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>The flannel is crashing whenever I tried to schedule pod creation.</p>

<h3>Background</h3>
<p>I think your issue is cased by your <code>2 Flannel CNI</code> pods <code>CrashLoopBackOff</code> status.</p>
<p>Your error</p>
<pre><code>Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff&quot; network for pod &quot;demo-6c59fb8f77-9x6sr&quot;: networkPlugin cni failed to set up pod &quot;demo-6c59fb8f77-9x6sr_default&quot; network: open /run/flannel/subnet.env: no such file or directory
</code></pre>
<p>is pointing that pod cannot be created due to lack of <code>/run/flannel/subnet.env</code> file.
In <a href="https://github.com/coreos/flannel#flannel" rel="noreferrer">Flannel Github</a> document you can find:</p>
<blockquote>
<p>Flannel runs a small, single binary agent called flanneld <strong>on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space.</strong></p>
</blockquote>
<p>Meaning, to proper work, <code>Flannel</code> pod should be running on each node as it contains subnets information. From your outputs I can see that <strong>only 1</strong> is working properly <strong>out of 3</strong> <code>Flannel</code> pods.</p>
<pre><code>NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
...
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
</code></pre>
<p>If mentioned pod was scheduled on node where flannel pod is not working it won't be created due to <code>CNI network issues</code>. Besides your <code>demo</code> pod, also <code>kubernetes-dashboard</code> pods have the same issue with <code>ContainerCreating</code> status.</p>
<h3>Conclusion</h3>
<p>Your <code>demo</code> pod cannot be scheduled as Kubernetes encounter some network issues related with flannel configuration file (<code>...network: open /run/flannel/subnet.env: no such file or directory</code>).</p>
<p>Your flannel pods restarts counts is very high as for <code>27 hours</code>. You have to determine why and fix it. It might be lack of resources, network issues with your infrastructure or many other reasons. Once all <code>flannel</code> pods will be working correctly, your shouldn't encounter this error.</p>
<h3>Solution</h3>
<p>You have to make flannel pods works correctly on each node.</p>
<h3>Additional Troubleshooting Details</h3>
<p>For detailed investigation please provide</p>
<pre><code>$ kubectl describe kube-flannel-ds-kqz4t -n kube-system
$ kubectl describe kube-flannel-ds-v9q47 -n kube-system
</code></pre>
<p>Logs details would be also helpful</p>
<pre><code>$ kubectl logs kube-flannel-ds-kqz4t -n kube-system
$ kubectl logs kube-flannel-ds-v9q47 -n kube-system
</code></pre>
<p>Please replace <code>kubectl get pods --all-namespaces</code> with <code>kubectl get pods -o wide -A</code> and output of <code>kubectl get nodes -o wide</code>.</p>
<p>If you will provide those information, it should be possible to determine root cause of <code>flannel</code> pods issues and I will edit this answer with exact solution.</p>
