title: GitLab CI in GKE private cluster can't connect to master
body: <p>So far we have been using GKE public cluster for all our workloads. We have created a second, private cluster (still GKE) with improved security and availability (old one is single zone, new one is regional cluster). We are using Gitlab.com for our code, but using self-hosted Gitlab CI runner in the clusters.</p>
  
  <p>The runner is working fine on the public cluster, all workloads complete successfully. However on the private cluster, all kubectl commands of thr CI fail with <code>Unable to connect to the server: dial tcp &lt;IP&gt;:443: i/o timeout error</code>. The CI configuration has not changed - same base image, still using gcloud SDK with a CI-specific service account to authenticate to the cluster.</p>
  
  <p>Both clusters have master authorized networks enabled and have only our office IPs are set. Master is accessible from a public IP. Authentication is successful, client certificate &amp; basic auth are disabled on both. Cloud NAT is configured, nodes have access to the Internet (can pull container images, Gitlab CI can connect etc).</p>
  
  <p>Am I missing some vital configuration? What else should I be looking at?</p>