<p>I am trying to further understand what exactly is happening when I provision a private cluster in Google's Kubernetes Engine.</p>
<p>Google provides this example here of provisioning a private cluster where the control plane services (e.g. Kubernetes API) live on the <code>172.16.0.16/28</code> subnet.</p>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters</a></p>
<pre><code>gcloud beta container clusters create pr-clust-1 \
 --private-cluster \
 --master-ipv4-cidr 172.16.0.16/28 \
 --enable-ip-alias \
 --create-subnetwork &quot;&quot;
</code></pre>
<p>When I run this command, I see that:</p>
<ul>
<li>I now have a few gke subnets in my VPC belong to the cluster subnets for nodes and services. These are in the <code>10.x.x.x/8</code> range.</li>
<li>I don't have any subnets in the <code>172.16/16</code> address space.</li>
<li>I do have some new pairing rules and routes that seem to be related.  For example, there is a new route <code>peering-route-a08d11779e9a3276</code> with a destination address range of <code>172.16.0.16/28</code> and next hop <code>gke-62d565a060f347e0fba7-3094-3230-peer</code>.  This peering role then points to <code>gke-62d565a060f347e0fba7-3094-bb01-net</code></li>
</ul>
<pre><code>gcloud compute networks subnets list | grep us-west1

#=&gt;

default                     us-west1                 default  10.138.0.0/20
gke-insti3-subnet-62d565a0  us-west1                 default  10.2.56.0/22
</code></pre>
<pre><code>gcloud compute networks peerings list

#=&gt;

NAME                                     NETWORK  PEER_PROJECT              PEER_NETWORK                                        AUTO_CREATE_ROUTES  STATE   STATE_DETAILS
gke-62d565a060f347e0fba7-3094-3230-peer  default  gke-prod-us-west1-a-4180  gke-62d565a060f347e0fba7-3094-bb01-net              True                ACTIVE  [2018-08-23T16:42:31.351-07:00]: Connected.
</code></pre>
<p>Is <code>gke-62d565a060f347e0fba7-3094-bb01-net</code> a peered VPC in which the Kubernetes management endpoints live (the control plane stuff in the <code>172.16/16</code> range) that Google is managing for the GKE service?</p>
<p>Further - how are my requests making it to the Kubernetes API server?</p>

Chris Mutzel
<p>The <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters" rel="noreferrer">Private Cluster</a> feature of GKE depends on the <a href="https://cloud.google.com/vpc/docs/alias-ip" rel="noreferrer">Alias IP Ranges</a> feature of VPC networking, so there are multiple things happening when you create a private cluster:</p>

<ul>
<li><p>The <code>--enable-ip-alias</code> flag tells GKE to use a subnetwork that has two secondary IP ranges: one for pods and one for services. This allows the VPC network to understand all the IP addresses in your cluster and route traffic appropriately.</p></li>
<li><p>The <code>--create-subnetwork</code> flag tells GKE to create a new subnetwork (gke-insti3-subnet-62d565a0 in your case) and choose its primary and secondary ranges automatically. Note that you could instead choose the secondary ranges yourself with <code>--cluster-ipv4-cidr</code> and <code>--services-ipv4-cidr</code>. Or you could even create the subnetwork yourself and tell GKE to use it with the flags <code>--subnetwork</code>, <code>--cluster-secondary-range-name</code>, and <code>--services-secondary-range-name</code>.</p></li>
<li><p>The <code>--private-cluster</code> flag tells GKE to create a new VPC network (gke-62d565a060f347e0fba7-3094-bb01-net in your case) in a Google-owned project and connect it to your VPC network using <a href="https://cloud.google.com/vpc/docs/vpc-peering" rel="noreferrer">VPC Network Peering</a>. The Kubernetes management endpoints live in the range you specify with <code>--master-ipv4-cidr</code> (172.16.0.16/28 in your case). An <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing" rel="noreferrer">Internal Load Balancer</a> is also created in the Google-owned project and this is what your worker nodes communicate with. This ILB allows traffic to be load-balanced across multiple VMs in the case of a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters" rel="noreferrer">Regional Cluster</a>. You can find this internal IP address as the <code>privateEndpoint</code> field in the output of <code>gcloud beta container clusters describe</code>. The important thing to understand is that all communication between master VMs and worker node VMs happens over internal IP addresses, thanks to the VPC peering between the two networks.</p></li>
<li><p>Your private cluster <strong>also</strong> has an external IP address, which you can find as the <code>endpoint</code> field in the output of <code>gcloud beta container clusters describe</code>. This is not used by the worker nodes, but is typically used by customers to manage their cluster remotely, <em>e.g.</em>, using <code>kubectl</code>.</p></li>
<li><p>You can use the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/authorized-networks" rel="noreferrer">Master Authorized Networks</a> feature to restrict which IP ranges (both internal and external) have access to the management endpoints. This feature is strongly recommended for private clusters, and is enabled by default when you create the cluster using the <code>gcloud</code> CLI.</p></li>
</ul>

<p>Hope this helps!</p>
