<p>I have 3 node Kubernetes cluster on 1.11 deployed with kubeadm and weave(CNI) running of version 2.5.1. I am providing weave CIDR of IP range of 128 IP's. After two reboot of nodes some of the pods stuck in <code>containerCreating</code> state. </p>

<p>Once you run <code>kubectl describe pod &lt;pod_name&gt;</code> you will see following errors:</p>

<pre><code>Events:
Type     Reason                  Age                From                Message
----     ------                  ----               ----                
-------
Normal   SandboxChanged          20m (x20 over 1h)  kubelet, 10.0.1.63  Pod sandbox changed, it will be killed and re-created.
Warning  FailedCreatePodSandBox  30s (x25 over 1h)  kubelet, 10.0.1.63  Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
</code></pre>

<p>If I check how many containers are running and how many IP address are allocated to those, I can see 24 containers:</p>

<pre><code>[root@ip-10-0-1-63 centos]# weave ps | wc -l
26
</code></pre>

<p>The number of total IP's to weave at that node is 42.</p>

<pre><code>[root@ip-10-0-1-212 centos]# kubectl exec -n kube-system -it weave-net-6x4cp -- /home/weave/weave --local status ipam
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-6x4cp -n kube-system' to see all of the containers in this pod.
6e:0d:f3:d7:f5:49(10.0.1.63)                42 IPs (32.8% of total) (42 active)
7a:24:6f:3c:1b:be(10.0.1.212)               40 IPs (31.2% of total) 
ee:00:d4:9f:9d:79(10.0.1.43)                46 IPs (35.9% of total) 
</code></pre>

<p>You can see all 42 IP's are active so no more IP's are available to allocate to new containers. But out of 42 only 26 are actually allocated to containers, I am not sure where are remaining IP's. It is happening on all three nodes.</p>

<p>Here is the output of weave status for your reference:</p>

<pre><code>[root@ip-10-0-1-212 centos]# weave status

    Version: 2.5.1 (version 2.5.2 available - please upgrade!)

    Service: router
    Protocol: weave 1..2
       Name: 7a:24:6f:3c:1b:be(10.0.1.212)
    Encryption: disabled
PeerDiscovery: enabled
    Targets: 3
Connections: 3 (2 established, 1 failed)
      Peers: 3 (with 6 established connections)
TrustedSubnets: none

    Service: ipam
     Status: waiting for IP(s) to become available
      Range: 192.168.13.0/25
DefaultSubnet: 192.168.13.0/25
</code></pre>

<p>If you need anymore information, I would happy to provide. Any Clue?</p>

Prafull Ladha
<p>Not sure if we have the same problem.
But before i reboot a node. I need to <code>drain</code> it first. So, all pods in that nodes will be evicted. We are safe to reboot the node.
After that node is up. You need to <code>uncordon</code> again. The node will be available to scheduling pod again.</p>

<p>My reference <a href="https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/</a></p>
