<p>I am trying to use <a href="https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html" rel="nofollow noreferrer">CNI Custom Networking on EKS</a> to make sure that Pod IPs are allocated from alternative subsets (to prevent IP starvation in the subnets my cluster nodes are running in). To do this I need to create some ENIConfigs and annotate each node.</p>

<p>How can I ensure that each node is annotated before any Pods are scheduled to it to ensure no Pod IPs are allocated from the subnets my nodes are running in?</p>

<p>EDIT: The only solution I can think of so far is:</p>

<ul>
<li>Add a NoSchedule taint to all nodes by default</li>
<li>Deploy a custom controller that tolerates the taint</li>
<li>Get the controller to annotate all nodes as required and remove the taint</li>
</ul>

<p>However, if the above is the only workaround that is a lot of effort for a managed service</p>

<p>How about:</p>

<ul>
<li>Add a <code>ENIConfigComplete: false</code> taint to all nodes by default</li>
<li>Deploy DaemonSet that tolerates <code>ENIConfigComplete: false</code></li>
<li>DaemonSet creates a pod on each new node which</li>
<li>creates some ENIConfigs on the node (bash script??)</li>
<li>annotates each node with <code>ENIConfigComplete: true</code></li>
<li>DaemonSet no longer tolerates the node, so</li>
<li>Pod is removed from the node.</li>
</ul>

<p>The DaemonSet would ensure that every new node was properly set up.</p>

<p>Salesforce talk about this technique for provisioning the disks on their new nodes:</p>

<ul>
<li><a href="https://engineering.salesforce.com/provisioning-kubernetes-local-persistent-volumes-61a82d1d06b0" rel="nofollow noreferrer">https://engineering.salesforce.com/provisioning-kubernetes-local-persistent-volumes-61a82d1d06b0</a></li>
</ul>

<p>It would avoid having a long running controller process.</p>
