<p>The <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noreferrer">Kubernetes Service doc</a> shows the below explanation of how Node security groups are updated for each NLB Service.</p>
<p><a href="https://i.stack.imgur.com/skGyr.png" rel="noreferrer"><img src="https://i.stack.imgur.com/skGyr.png" alt="enter image description here" /></a></p>
<p>Unfortunately, I have a VPC that has 3 different CIDRs.  This means that for every port on a Service, 4 new rules are added to the Nodes' security group.  There is a team that has a NLB Service with 5 ports, which means it results in 20 new rules added to the Nodes' security group.  Other teams normally have 2 Ports, which results in 8 rules added to the Nodes' security group.  The end result is we sometimes reach the max amount of 64 Rules allowed on one Security Group.</p>
<p>What are ideas to design around this so that teams can create as many NLB Services with as many ports as they want?</p>

<p>The <a href="https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html#network-load-balancer" rel="nofollow noreferrer">EKS documentation</a> says something about this.</p>
<blockquote>
<p>For each NLB that you create Amazon EKS adds one inbound rule to the
node's security group for client traffic and one rule for each load
balancer subnet in the VPC for health checks. Deployment of a service
of type LoadBalancer can fail if Amazon EKS attempts to create rules
that exceed the quota for the maximum number of rules allowed for a
security group. For more information, see Security groups in Amazon
VPC quotas in the Amazon VPC User Guide. Consider the following
options to minimize the chances of exceeding the maximum number of
rules for a security group.</p>
<ul>
<li><p>Request an increase in your rules per security group quota. For more
information, see Requesting a quota increase in the Service Quotas
User Guide.</p>
</li>
<li><p>Use Create a network load balancer, rather than instance targets. With
IP targets, rules can potentially be shared for the same target ports.
Load balancer subnets can be manually specified with an annotation.
For more information, see Annotations on GitHub.</p>
</li>
<li><p>Use an Ingress, instead of a Service of type LoadBalancer to send
traffic to your service. The AWS Application Load Balancer (ALB)
requires fewer rules than NLBs. An ALB can also be shared across
multiple Ingresses. For more information, see Application load
balancing on Amazon EKS.</p>
</li>
<li><p>Deploy your clusters to multiple accounts.</p>
</li>
</ul>
</blockquote>
<p>If none of those options work for you, and you have also ALBs, you can minimize the rules for those ALB forcing them to use a specific security group instead of adding their rules to the node's security group. The annotation is <code>service.beta.kubernetes.io/aws-load-balancer-security-groups</code>. Doing so, you replace several rules for just one, leaving more space for NLBs rules.</p>
