<p>I have a simple pod in a namespace called <code>"a"</code>, 
and another pod in a namespace <code>"b"</code>...</p>

<p>I also have a test script that makes a grpc call from <code>"a"</code> to <code>"b"</code>.</p>

<ol>
<li>This script <strong>does not work</strong> in using the FQDN (e.g. "some-service.b.cluster.local")</li>
<li>This script <strong>does work</strong> if using the exact IP Address of the pod in "b" namespace</li>
</ol>

<p>I suppose is some error in DNS Resolution, 
but I can't really move forward.</p>

<p>Any help?</p>

<pre><code>kubectl exec -n a somepod-f647b7d95-mrvfr cat /etc/resolv.conf

nameserver 10.12.0.10
search chimera.svc.cluster.local svc.cluster.local cluster.local c.&lt;company-name&gt;-staging.internal &lt;provider&gt;.internal
options ndots:5
</code></pre>

<pre><code>kubectl get pods -n kube-system

event-exporter-v0.2.4-6d4c69fbfb-f4xpf                           1/1     Running   0          24d
fluentd-gcp-scaler-6965bb45c9-mzvw6                              1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-2m2bf                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-2v6bq                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-4xpbc                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-7g5hm                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-8mqvc                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-f9hrs                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-fr58c                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-hzrsb                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-kq8hc                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-kt6p5                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-nsztm                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-qcl4r                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-qggv9                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-qkkp5                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-rm9hn                                         1/1     Running   0          5d5h
fluentd-gcp-v3.2.0-sv52h                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-t75fp                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-v49fv                                         1/1     Running   0          7d6h
kube-dns-6cd7bbdf65-jnntn                                        4/4     Running   0          24d
kube-dns-6cd7bbdf65-txmlj                                        4/4     Running   0          24d
kube-dns-autoscaler-8687c64fc-29jgq                              1/1     Running   0          7d6h
kube-proxy-gke-iceberg-api-v2-201908101259587443-01f0b55b-q0k3   1/1     Running   0          217d
kube-proxy-gke-iceberg-api-v2-201908101259587443-0d661dfb-3zhx   1/1     Running   0          217d
kube-proxy-gke-iceberg-api-v2-201908101259587443-92bbd393-w96w   1/1     Running   1          115d
kube-proxy-gke-iceberg-es-single-202003021919386-1b520a2e-sn9m   1/1     Running   0          5d6h
kube-proxy-gke-iceberg-es-single-202003021919386-bf6046bf-7wsp   1/1     Running   0          5d5h
kube-proxy-gke-iceberg-es-single-202003021919386-d64daa4e-1jqz   1/1     Running   0          5d5h
kube-proxy-gke-iceberg-general-20190810125958886-21ed2623-4m0p   1/1     Running   0          217d
kube-proxy-gke-iceberg-general-20190810125958886-8b185cf9-x1j2   1/1     Running   0          217d
kube-proxy-gke-iceberg-general-20190810125958886-eaf63d3c-k338   1/1     Running   0          217d
kube-proxy-gke-iceberg-kafka-2019081012595876540-429586da-m2qf   1/1     Running   0          217d
kube-proxy-gke-iceberg-kafka-2019081012595876540-76ebb654-z7xx   1/1     Running   0          217d
kube-proxy-gke-iceberg-kafka-2019081012595876540-c3abee6e-4q76   1/1     Running   0          217d
kube-proxy-gke-iceberg-rabbitmq-2019081012595876-552d6676-8z2k   1/1     Running   0          217d
kube-proxy-gke-iceberg-rabbitmq-2019081012595876-662980f7-76jc   1/1     Running   0          217d
kube-proxy-gke-iceberg-rabbitmq-2019081012595876-b269df22-6zqj   1/1     Running   0          217d
kube-proxy-gke-iceberg-redis-2019081012595877180-38264a5e-c0ch   1/1     Running   0          217d
kube-proxy-gke-iceberg-redis-2019081012595877180-9412d5f5-pt3w   1/1     Running   0          217d
kube-proxy-gke-iceberg-redis-2019081012595877180-947dc20b-c002   1/1     Running   0          217d
kube-state-metrics-67b67d8fdd-nkpt4                              2/2     Running   0          24d
l7-default-backend-fd59995cd-cvqwb                               1/1     Running   0          24d
metrics-server-v0.3.1-5c8f664b95-sthjz
</code></pre>

Hitmands
<p>From your description it appears you're running KubeDNS. My First piece of advice to you would be to <a href="https://coredns.io/2018/05/21/migration-from-kube-dns-to-coredns/" rel="nofollow noreferrer">migrate</a> to CoreDNS as KubeDNS is on a <a href="https://github.com/kubernetes/kubeadm/issues/1943" rel="nofollow noreferrer">deprecation path</a>.</p>

<p>Secondly, two things leap out at me.</p>

<ol>
<li><p>You're talking about making calls between pods instead of <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">services</a>. While Kubernetes does provide Service Discovery between your applications, it does this through DNS as you know. However, just because pods can resolve each other <a href="https://stackoverflow.com/a/53517878/3066788">doesn't mean</a> a container will have its ports exposed outside of its pod. To do this, even to an application within the cluster than can resolve it, you must declare a Service resource for each Pod or Controller. </p></li>
<li><p>When you talk about making a call that references a FQDN of your B pod/service, you do not specify the default FQDN schema and do not mention having customized this. </p></li>
</ol>

<p>First, can you please <code>kubectl get svc -n NAMESPACE</code> for both namespaces in which your A and B pods are running and confirm that a Service of type <a href="https://stackoverflow.com/questions/41509439/whats-the-difference-between-clusterip-nodeport-and-loadbalancer-service-types">ClusterIP</a> has been created and an IP Address has been associated with the Service?</p>

<p>Secondly, can you try and perform a connection attempt from Application A to the Service of Application B by <a href="https://stackoverflow.com/a/55650127/3066788">specifying the following FQDN format</a> instead?</p>

<p><code>some-service.b.svc.cluster.local</code></p>

<p>Notice the <strong>svc</strong> portion. You mentioned <code>some-service.b.cluster.local</code> in your OP.</p>

<p>Lastly, if everything comes back normal we can start troubleshooting <code>kube-dns</code>. it does appear that all three pods are running. However, have you tried to <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application-introspection/" rel="nofollow noreferrer">describe</a> them and/or <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="nofollow noreferrer">grab their logs</a>? Could you try the following and share a summary if anything looks interesting?</p>

<pre><code>kubectl describe pod -n kube-system kube-dns-6cd7bbdf65-jnntn
kubectl describe pod -n kube-system kube-dns-6cd7bbdf65-txmlj
kubectl describe pod -n kube-system kube-dns-autoscaler-8687c64fc-29jgq
</code></pre>

<pre><code>kubectl logs -n kube-system kube-dns-6cd7bbdf65-jnntn
kubectl logs -n kube-system kube-dns-6cd7bbdf65-txmlj
kubectl logs -n kube-system kube-dns-autoscaler-8687c64fc-29jgq
</code></pre>

<p>I imagine the <code>logs</code> commands will provide you with the answer you're looking for. Let me know if you need any further clarification or help on this issue. I'm happy to help. </p>
