<p>Multiple pods of a 600 pod deployment stuck in <code>ContainerCreating</code> after a rolling update with the message:</p>

<blockquote>
  <p>Failed create pod sandbox: rpc error: code = Unknown desc =
  NetworkPlugin cni failed to set up pod  network: add cmd:
  failed to assign an IP address to container</p>
</blockquote>

<p>What I have tried:</p>

<ul>
<li>Upgraded to v1.12 on EKS and CNI 1.5.0.  <a href="https://github.com/aws/amazon-vpc-cni-k8s/issues/123" rel="noreferrer">This issue was closed stating CNI 1.5.0 solved the issue.</a>  It did not for us.  In another thread <a href="https://github.com/aws/amazon-vpc-cni-k8s/issues/69" rel="noreferrer">leaking ENIs was blamed</a> but was also closed due to CNI upgrade.</li>
<li><a href="https://aws.amazon.com/blogs/opensource/cni-metrics-helper/" rel="noreferrer">Installed cni-metrics-helper</a> and this is a snapshot of the output:</li>
</ul>

<pre><code>maxIPAddresses, value: 759.000000
ipamdActionInProgress, value: 1.000000
addReqCount, value: 16093.000000
awsAPILatency, value: 564.000000
delReqCount, value: 32337.000000
eniMaxAvailable, value: 69.000000
assignIPAddresses, value: 558.000000
totalIPAddresses, value: 682.000000
eniAllocated, value: 69.000000
</code></pre>

<p>Do the CNI metrics output suggest there's an issue?  Seems like there are enough IPs.</p>

<p>What else can I try to debug?</p>

<p>It seems that you reached maximum number of IP addresses in your subnet what can suggest such thing in <a href="https://github.com/liwenwu-amazon/amazon-vpc-cni-k8s-1/blob/metrics1/docs/troubleshooting.md#tip-make-sure-there-are-enough-enis-and-ips-for-pods-in-the-cluster" rel="noreferrer">documentation</a>:</p>

<blockquote>
  <p>maxIPAddress: the maximum number of IP addresses that can be used for
  Pods in the cluster. (assumes there is enough IPs in the subnet).</p>
</blockquote>

<p>Please take a look also on <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-unavailable" rel="noreferrer">maxUnavailable</a> and <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-surge" rel="noreferrer">maxSurge</a> parameters which controls how many PODs appear during rolling upgrade - maybe your configuration assumes that during rolling upgrade you will have over 600 PODs (like 130%) and that hit limits of your AWS network.</p>
