<p>We have setup a GKE cluster using Terraform with private and shared networking:</p>
<p>Network configuration:</p>
<pre><code>resource &quot;google_compute_subnetwork&quot; &quot;int_kube02&quot; {
  name          = &quot;int-kube02&quot;
  region        = var.region
  project       = &quot;infrastructure&quot;
  network       = &quot;projects/infrastructure/global/networks/net-10-23-0-0-16&quot;
  ip_cidr_range = &quot;10.23.5.0/24&quot;
  secondary_ip_range {
    range_name    = &quot;pods&quot;
    ip_cidr_range = &quot;10.60.0.0/14&quot; # 10.60 - 10.63
  }
  secondary_ip_range {
    range_name    = &quot;services&quot;
    ip_cidr_range = &quot;10.56.0.0/16&quot;
  }
}
</code></pre>
<p>Cluster configuration:</p>
<pre><code>resource &quot;google_container_cluster&quot; &quot;gke_kube02&quot; {
  name     = &quot;kube02&quot;
  location = var.region

  initial_node_count = var.gke_kube02_num_nodes

  network    = &quot;projects/ninfrastructure/global/networks/net-10-23-0-0-16&quot;
  subnetwork = &quot;projects/infrastructure/regions/europe-west3/subnetworks/int-kube02&quot;

  master_authorized_networks_config {
    cidr_blocks {
      display_name = &quot;admin vpn&quot;
      cidr_block   = &quot;10.42.255.0/24&quot;
    }
    cidr_blocks {
      display_name = &quot;monitoring server&quot;
      cidr_block   = &quot;10.42.4.33/32&quot;
    }
    cidr_blocks {
      display_name = &quot;cluster nodes&quot;
      cidr_block   = &quot;10.23.5.0/24&quot;
    }
  }

  ip_allocation_policy {
    cluster_secondary_range_name  = &quot;pods&quot;
    services_secondary_range_name = &quot;services&quot;
  }

  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = true
    master_ipv4_cidr_block  = &quot;192.168.23.0/28&quot;


  }

  node_config {
    machine_type = &quot;e2-highcpu-2&quot;

    tags = [&quot;kube-no-external-ip&quot;]
    metadata = {
      disable-legacy-endpoints = true
    }

    oauth_scopes = [
      &quot;https://www.googleapis.com/auth/logging.write&quot;,
      &quot;https://www.googleapis.com/auth/monitoring&quot;,
    ]
  }
}
</code></pre>
<p>The cluster is online and running fine. If I connect to one of the worker nodes i can reach the api using <code>curl</code>:</p>
<pre><code>curl -k https://192.168.23.2
{
  &quot;kind&quot;: &quot;Status&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;metadata&quot;: {

  },
  &quot;status&quot;: &quot;Failure&quot;,
  &quot;message&quot;: &quot;forbidden: User \&quot;system:anonymous\&quot; cannot get path \&quot;/\&quot;&quot;,
  &quot;reason&quot;: &quot;Forbidden&quot;,
  &quot;details&quot;: {

  },
  &quot;code&quot;: 403
}
</code></pre>
<p>I also see a healthy cluster when using a SSH port forward:</p>
<pre><code>❯ k get pods --all-namespaces --insecure-skip-tls-verify=true
NAMESPACE     NAME                                               READY   STATUS    RESTARTS   AGE
kube-system   event-exporter-gke-5479fd58c8-mv24r                2/2     Running   0          4h44m
kube-system   fluentbit-gke-ckkwh                                2/2     Running   0          4h44m
kube-system   fluentbit-gke-lblkz                                2/2     Running   0          4h44m
kube-system   fluentbit-gke-zglv2                                2/2     Running   4          4h44m
kube-system   gke-metrics-agent-j72d9                            1/1     Running   0          4h44m
kube-system   gke-metrics-agent-ttrzk                            1/1     Running   0          4h44m
kube-system   gke-metrics-agent-wbqgc                            1/1     Running   0          4h44m
kube-system   kube-dns-697dc8fc8b-rbf5b                          4/4     Running   5          4h44m
kube-system   kube-dns-697dc8fc8b-vnqb4                          4/4     Running   1          4h44m
kube-system   kube-dns-autoscaler-844c9d9448-f6sqw               1/1     Running   0          4h44m
kube-system   kube-proxy-gke-kube02-default-pool-2bf58182-xgp7   1/1     Running   0          4h43m
kube-system   kube-proxy-gke-kube02-default-pool-707f5d51-s4xw   1/1     Running   0          4h43m
kube-system   kube-proxy-gke-kube02-default-pool-bd2c130d-c67h   1/1     Running   0          4h43m
kube-system   l7-default-backend-6654b9bccb-mw6bp                1/1     Running   0          4h44m
kube-system   metrics-server-v0.4.4-857776bc9c-sq9kd             2/2     Running   0          4h43m
kube-system   pdcsi-node-5zlb7                                   2/2     Running   0          4h44m
kube-system   pdcsi-node-kn2zb                                   2/2     Running   0          4h44m
kube-system   pdcsi-node-swhp9                                   2/2     Running   0          4h44m
</code></pre>
<p>So far so good. Then I setup the Cloud Router to announce the <code>192.168.23.0/28</code> network. This was successful and replicated to our local site using BGP. Running <code>show route 192.168.23.2</code> displays the correct route is advertised and installed.</p>
<p>When trying to reach the API from the monitoring server <code>10.42.4.33</code> I just run into timeouts. All three, the Cloud VPN, the Cloud Router and the Kubernetes Cluster run in <code>europe-west3</code>.</p>
<p>When i try to ping one of the workers its working completely fine, so networking in general works:</p>
<pre><code>[me@monitoring ~]$ ping 10.23.5.216
PING 10.23.5.216 (10.23.5.216) 56(84) bytes of data.
64 bytes from 10.23.5.216: icmp_seq=1 ttl=63 time=8.21 ms
64 bytes from 10.23.5.216: icmp_seq=2 ttl=63 time=7.70 ms
64 bytes from 10.23.5.216: icmp_seq=3 ttl=63 time=5.41 ms
64 bytes from 10.23.5.216: icmp_seq=4 ttl=63 time=7.98 ms
</code></pre>
<p>Googles <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters" rel="nofollow noreferrer">Documentation</a> gives no hit what could be missing. From what I understand the Cluster API should be reachable by now.</p>
<p>What could be missing and why is the API not reachable via VPN?</p>

<p>I have been missing the peering configuration documented here:
<a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#cp-on-prem-routing" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#cp-on-prem-routing</a></p>
<pre><code>resource &quot;google_compute_network_peering_routes_config&quot; &quot;peer_kube02&quot; {
  peering = google_container_cluster.gke_kube02.private_cluster_config[0].peering_name
  project = &quot;infrastructure&quot;
  network = &quot;net-10-13-0-0-16&quot;

  export_custom_routes = true
  import_custom_routes = false
}

</code></pre>
