<p>We are trying to inter communicate with multiple services without exposing them to the public.</p>
<p>I have a service like this:-</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: &quot;v1&quot;
kind: &quot;Service&quot;
metadata:
  name: &quot;config&quot;
  namespace: &quot;noicenamespace&quot;
  labels:
    app: &quot;config&quot;
spec:
  ports:
  - protocol: &quot;TCP&quot;
    port: 8888
    targetPort: 8888
  selector:
    app: &quot;config&quot;
  type: &quot;LoadBalancer&quot;
  loadBalancerIP: &quot;&quot;

</code></pre>
<p>Due to type LoadBalancer, the service is accesible on public network. But we only want this service to be visible to our internal services in our cluster.</p>
<p>So if I comment out the loadBalancerIP and set type as ClusterIP, My other pods can't access the service.
I tried specifying the service name like this:-</p>
<pre><code>http://config.noicenamespace.svc.cluster.local:8888
</code></pre>
<p>But I get timeout. We have created cluster from scratch on Google Kubernetes Engine</p>

LightSith
<p>This error <code>&quot;Error from server: error dialing backend: dial timeout&quot;</code> is related to the progressive introduction of Konnectivity network proxy in some clusters starting from GKE 1.19.</p>
<p>The Konnectivity network proxy (KNP) provides a TCP level proxy for master egress (kube-apiserver to cluster communication),</p>
<p>The Konnectivity service consists of two parts: the Konnectivity server in the control plane network and the Konnectivity agents in the nodes network. The Konnectivity agents initiate connections to the Konnectivity server and maintain the network connections.
After enabling the Konnectivity service, all control plane to nodes traffic goes through these connections, due to this there must be a firewall rule that allows communication correctly through port(use the port number getting displayed in the error message with the IP of the end point), otherwise dial timeout errors may occur.</p>
<p>By using this filter on Cloud Logging you can found the error logs related to konnectivity agent timeout connection due to missing firewall rule:(Note the IP address and the Port number of the endpoint from the error use the details in the firewall rule)</p>
<pre><code>resource.labels.cluster_name=&quot;cluster name&quot;
&quot;konnectivity-agent&quot;
</code></pre>
<p>Add a firewall egress rule that allow you to connect to the port (use the port number getting displayed in the error message with the IP of the end point) , you could use the following command to add that rule. that should allow the konnectivity-agent to connect to the control plane.</p>
<pre><code>gcloud compute firewall-rules create gke-node-to-konnectivity-service \
--allow=tcp:&lt;port number&gt; \
--direction=EGRESS \
--destination-ranges=&lt;endpoint IP address &gt; \
--target-tags=&lt; node name&gt; \
--priority=1000
</code></pre>
