<p>I've setup a VPC consisting of a public subnet and multiple private subnets, the public subnet hosts a OpenVpn access server through which I can access my instances running in private subnets, I've all the NAT and Internet Gateway running fine and I can access internet from instances running in private subnet over VPN.</p>

<p>Everything was running fine until I decided to run a EKS instances in one of my private subnet with "Public Access" feature disabled, I cannot reach to my EKS endpoint (K8 API service endpoint API) over VPN or from any instances running into my public/private subnets (i.e. using a Jump box).</p>

<p>I googled a lot and found that I've to enable <code>enableDnsHostnames</code> and <code>enableDnsSupport</code> with my VPC, but enabling these did not help. I also checked my Master node security group which allows an inbound traffic from anywhere i.e. <code>0.0.0.0/0</code> over port 443, so security group is not a concern.</p>

<p>However everything runs just fine if I turn on "Public Access" flag to <code>Enabled</code> but that fails purpose of creating K8 cluster in private subnet.</p>

<p>Can someone please point out where I'm mistaking, thanks in advance.</p>

<h3>Intro</h3>

<p>If you are setting up an EKS Kubernetes cluster on AWS then you would probably want a cluster that is not accessible to the world, then you'll access it privately via a VPN. Considering all disclosed <a href="https://www.cvedetails.com/vulnerability-list/vendor_id-15867/product_id-34016/Kubernetes-Kubernetes.html" rel="nofollow noreferrer">vulnerabilities</a> this setup looks as a more secure design, and it allows you to isolate the Kubernetes control plane and worker nodes within your VPC, providing an additional layer of protection to harden clusters against malicious attack and accidental exposure. </p>

<p>You do that by toggling off <em>Public Access</em> while creating the cluster, however a problem with that is the <a href="https://github.com/aws/containers-roadmap/issues/221" rel="nofollow noreferrer">automated DNS Resolution for EKS with private endpoint</a> is still not supported as it is, for example with RDS private endpoints.</p>

<p>When creating your EKS cluster:</p>

<ul>
<li>AWS does not allow you to change the DNS name of the endpoint.</li>
<li>AWS creates a managed private hosted zone for the endpoint DNS (not editable).</li>
</ul>

<h3>Solution N°1</h3>

<p>One suggested solution is to create Route53 inbound and outbound endpoints as described in this <a href="https://aws.amazon.com/blogs/compute/enabling-dns-resolution-for-amazon-eks-cluster-endpoints/" rel="nofollow noreferrer">official AWS blog post</a>. </p>

<p>However, the problem with that is that every time you create a cluster you will need to add IPs to our local resolver and If your local infrastructure is maintained by someone else then it might take days to get that done.</p>

<h3>Solution N°2</h3>

<p>You could solve that problem by writing a small script that updates <code>/etc/hosts</code> with the <strong>IP</strong> and <strong>dns</strong> name of the EKS private endpoint. This is kind of a hack but works well. </p>

<p>Here’s how the <code>eks-dns.sh</code> script looks:</p>

<pre class="lang-sh prettyprint-override"><code>#!/bin/bash

#
# eg: bash ~/.aws/eks-dns.sh bb-dev-eks-BedMAWiB bb-dev-devops
#
clusterName=$1
awsProfile=$2

#
# Get EKS ip addrs
#
ips=`aws ec2 describe-network-interfaces --profile $awsProfile \
--filters Name=description,Values="Amazon EKS $clusterName" \
| grep "PrivateIpAddress\"" | cut -d ":" -f 2 |  sed 's/[*",]//g' | sed 's/^\s*//'| uniq`

echo "#-----------------------------------------------------------------------#"
echo "# EKS Private IP Addresses:                                              "
echo $ips
echo "#-----------------------------------------------------------------------#"
echo ""

#
# Get EKS API endpoint
#
endpoint=`aws eks describe-cluster --profile $awsProfile --name $clusterName \
| grep endpoint\" | cut -d ":" -f 3 | sed 's/[\/,"]//g'`

echo "#-----------------------------------------------------------------------#"
echo "# EKS Private Endpoint                                                   "
echo $endpoint
echo "#-----------------------------------------------------------------------#"
echo ""

IFS=$'\n'
#
# Create backup of /etc/hosts
#
sudo cp /etc/hosts /etc/hosts.backup.$(date +%Y-%m-%d)

#
# Clean old EKS endpoint entries from /etc/hots
#
if grep -q $endpoint /etc/hosts; then
  echo "Removing old EKS private endpoints from /etc/hosts"
  sudo sed -i "/$endpoint/d" /etc/hosts
fi

#
# Update /etc/hosts with EKS entry
#
for item in $ips
do
    echo "Adding EKS Private Endpoint IP Addresses"
    echo "$item $endpoint" | sudo tee -a /etc/hosts
done
</code></pre>

<h3>Exec Example</h3>

<pre><code>╭─delivery at delivery-I7567 in ~ using ‹› 19-11-21 - 20:26:27
╰─○ bash ~/.aws/eks-dns.sh bb-dev-eks-BedMAWiB bb-dev-devops
</code></pre>

<p>Resulting <code>/etc/hosts</code></p>

<pre><code>╭─delivery at delivery-I7567 in ~ using ‹› 19-11-21 - 20:26:27
╰─○ cat /etc/hosts
127.0.0.1       localhost 

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

172.18.3.111 D4EB2912FDB14E8DAB358D471DD0DC5B.yl4.us-east-1.eks.amazonaws.com
172.18.1.207 D4EB2912FDB14E8DAB358D471DD0DC5B.yl4.us-east-1.eks.amazonaws.com
</code></pre>

<ul>
<li><strong>Ref Article:</strong> studytrails.com/devops/kubernetes/local-dns-resolution-for-eks-with-private-endpoint/</li>
</ul>

<h3>Important Consideration</h3>

<p>Also as stated in the related Question: <a href="https://stackoverflow.com/questions/55510783/cant-access-eks-api-server-endpoint-within-vpc-when-private-access-is-enabled?rq=1">Can&#39;t access EKS api server endpoint within VPC when private access is enabled</a> , your VPC must have <code>enableDnsHostnames</code> and <code>enableDnsSupport</code> set to true.</p>

<blockquote>
  <p>I had to enable enableDnsHostnames and enableDnsSupport for my VPC.</p>
  
  <p>When enabling the private access of a cluster, EKS creates a private
  hosted zone and associates with the same VPC. It is managed by AWS
  itself and you can't view it in your aws account. So, this private
  hosted zone to work properly, your VPC must have enableDnsHostnames
  and enableDnsSupport set to true.</p>
  
  <p>Note: Wait for a while for changes to be reflected(about 5 minutes).</p>
</blockquote>
