<p>I am new to AWS, Kubernetes, EKS, and AppMesh, but had done DevOps in previous roles.</p>
<p>I am taking over some K8 cluster which used EKS and found that we set up NAT gateway which helps redirect egress traffic outbound as a single IP (we need that for whitelisting as 3rd-party external service require it).  Pods hosted in a private subnet works fine.</p>
<p>But I found that Pods which hosted on public subnet just skip the NAT gateway, it uses the Public DNS (IPv4) IP address for outbound calls, which don't work for us as it does not use the single NAT gateway.</p>
<p>So I have a few questions:</p>
<ol>
<li>How do we migrate Pods from Public subnet Hosts to Private subnets Hosts?</li>
<li>Should we use <code>nodeSelector</code>, <code>Node affinity</code>? Do labelings on the Nodes work?</li>
<li>I am not sure why we have Nodes in a public subnet, but we followed this guide: <a href="https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html</a></li>
<li>If we do choose to be on fully private subnets, can we make an exception for such mapping that we allow some Pods to have HTTP endpoints to be exposed for ingress traffic, while still on private subnets?</li>
<li>What do you recommend us to handle when a Pod/Container needs to use NAT gateway for egress traffic, but then exposing HTTP endpoints for ingress traffic?</li>
</ol>
<p><a href="https://i.stack.imgur.com/b6EJX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/b6EJX.png" alt="" /></a></p>
<p>Note that currently, our EKS is by default set to all public, should we move to Public and private mode?</p>
<p><a href="https://i.stack.imgur.com/HcqW7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HcqW7.png" alt="" /></a></p>
<p>Thanks for all the answers in advance!.</p>

samxiao
<blockquote>
<p>How do we migrate Pods from Public subnet Hosts to Private subnets Hosts? Should we use nodeSelector, Node affinity? Do labelings on the Nodes work?</p>
</blockquote>
<p>Yes. Use <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">Node Affinity</a> which same as using a nodeSelector. You can do a rolling change by updating whatever resource you are using to manage your pods (i.e Deployment, Statefulset, DaemonSet, etc). If you configured it correctly the next time your pods start, they will be in the private subnet hosts.</p>
<blockquote>
<p>I am not sure why we have Nodes in a public subnet, but we followed this guide:
<a href="https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html</a></p>
</blockquote>
<p>The guide says public subnet so it makes sense that there is one.</p>
<blockquote>
<p>If we do choose to be on fully private subnets, can we make an exception for such mapping that we allow some Pods to have HTTP endpoints to be exposed for ingress traffic, while still on private subnets?</p>
</blockquote>
<p>Yes! you can create an externally facing load balancer (ALB, NLB, or ELB). These can also be managed by Kubernetes if you use the Service type <a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer" rel="nofollow noreferrer">LoadBalancer</a>. You'll need the <a href="https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html" rel="nofollow noreferrer">appropriate annotations</a> in your Service definitions to get what you want.</p>
<blockquote>
<p>What do you recommend us to handle when a Pod/Container needs to use NAT gateway for egress traffic, but then exposing HTTP endpoints for ingress traffic?</p>
</blockquote>
<p>Use an externally facing load balancer that forwards traffic to your private VPC with the Kubernetes Service type LoadBalancer and use <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html" rel="nofollow noreferrer">AWS NAT Gateways</a> for outgoing internet traffic.</p>
<p>Disclaimer: This is just a recommendation, there are other combinations and alternatives.</p>
