<p>So far we have been using GKE public cluster for all our workloads. We have created a second, private cluster (still GKE) with improved security and availability (old one is single zone, new one is regional cluster). We are using Gitlab.com for our code, but using self-hosted Gitlab CI runner in the clusters.</p>

<p>The runner is working fine on the public cluster, all workloads complete successfully. However on the private cluster, all kubectl commands of thr CI fail with <code>Unable to connect to the server: dial tcp &lt;IP&gt;:443: i/o timeout error</code>. The CI configuration has not changed - same base image, still using gcloud SDK with a CI-specific service account to authenticate to the cluster.</p>

<p>Both clusters have master authorized networks enabled and have only our office IPs are set. Master is accessible from a public IP. Authentication is successful, client certificate &amp; basic auth are disabled on both. Cloud NAT is configured, nodes have access to the Internet (can pull container images, Gitlab CI can connect etc).</p>

<p>Am I missing some vital configuration? What else should I be looking at?</p>

<p>I have found the solution to my problem, but I am not fully sure of the cause.</p>

<p>I used <code>gcloud container clusters get-credentials [CLUSTER_NAME]</code>, which gave the master's public endpoint. However that is inaccessible from within the cluster for some reason - so I assume it would require adding the public IP of the NAT (which is not statically provided) to the authorized networks.</p>

<p>I added the --internal-ip flag, which gave the cluster's internal IP address. The CI is now able to connect to the master.</p>

<p>Source: <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#internal_ip" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#internal_ip</a></p>

<p>tl;dr - <code>gcloud container clusters get-credentials --internal-ip [CLUSTER_NAME]</code> </p>
