<p>I have two Microservices deployed on K8S cluster (Locally on 3 VMs - 1 Master and 2 Worker Nodes):<br>
1- currency-exchange Microservice<br>
2- currency-conversion Microservice<br></p>
<p>I am trying to call <strong>currency-exchange</strong> Microservice from <strong>currency-conversion</strong> by using service name : <br>
http:///currency-exchange:8000.</p>
<p>It returns error as below: <br>
<code>{&quot;timestamp&quot;:&quot;2021-02-17T08:38:25.590+0000&quot;,&quot;status&quot;:500,&quot;error&quot;:&quot;Internal Server Error&quot;,&quot;message&quot;:&quot;currency-exchange executing GET http://currency-exchange:8000/currency-exchange/from/EUR/to/INR&quot;,&quot;path&quot;:&quot;/currency-conversion/from/EUR/to/INR/quantity/10&quot;}</code></p>
<p>I am using Kubernetes, CentOS8 using Calico CNI with set FELIX_IPTABLESBACKEND=NFT , based on <a href="https://stackoverflow.com/questions/65836764/kubernetes-ingress-second-node-port-is-not-responding">this link</a> to facilitate POD-TO-POD communications. <br>
Current services available:</p>
<pre><code>[root@k8s-master ~]# kubectl get svc
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE

currency-conversion                  NodePort    10.106.70.108    &lt;none&gt;        8100:32470/TCP               3h40m

currency-exchange                    NodePort    10.110.232.189   &lt;none&gt;        8000:31776/TCP               3h41m
</code></pre>
<p>Pods:<br></p>
<pre><code>[root@k8s-master ~]# kubectl get pods -o wide
NAME                                        READY   STATUS    RESTARTS   AGE     IP                NODE            NOMINATED NODE   READINESS GATES

currency-conversion-86d9bc4698-rxdkh        1/1     Running   0          5h45m   192.168.212.125   worker-node-1   &lt;none&gt;           &lt;none&gt;
currency-exchange-c79ff888b-c8sdd           1/1     Running   0          5h44m   192.168.19.160    worker-node-2   &lt;none&gt;           &lt;none&gt;
currency-exchange-c79ff888b-nfqpx           1/1     Running   0          5h44m   192.168.212.65    worker-node-1   &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>List of CoreDNS Pods available:<br></p>
<pre><code>[root@k8s-master ~]# kubectl get pods -o wide -n kube-system | grep coredns
coredns-74ff55c5b-9x5qm                    1/1     Running   8          25d   192.168.235.218   k8s-master      &lt;none&gt;           &lt;none&gt;
coredns-74ff55c5b-zkkn7                    1/1     Running   8          25d   192.168.235.220   k8s-master      &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>List all ENV variables:<br></p>
<pre><code>[root@k8s-master ~]# kubectl exec -it currency-conversion-86d9bc4698-rxdkh -- printenv

HOSTNAME=currency-conversion-86d9bc4698-rxdkh
CURRENCY_EXCHANGE_SERVICE_HOST=http://currency-exchange
KUBERNETES_SERVICE_HOST=10.96.0.1
CURRENCY_EXCHANGE_SERVICE_PORT=8000

........
</code></pre>
<p>nslookup kubernetes.default exec Command :<br></p>
<pre><code>[root@k8s-master ~]# kubectl exec -it currency-conversion-86d9bc4698-rxdkh -- nslookup kubernetes.default
    nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'kubernetes.default': Try again
command terminated with exit code 1
</code></pre>
<p>How do people solve such a problem? do they configure/tweak DNS to work properly as service registry ?</p>
<p>Thanks in advance</p>
<p><strong>EDITED:</strong></p>
<pre><code>[root@k8s-master ~]# kubectl describe service currency-conversion
Name:                     currency-conversion
Namespace:                default
Labels:                   app=currency-conversion
Annotations:              &lt;none&gt;
Selector:                 app=currency-conversion
Type:                     NodePort
IP Families:              &lt;none&gt;
IP:                       10.106.70.108
IPs:                      10.106.70.108
Port:                     &lt;unset&gt;  8100/TCP
TargetPort:               8100/TCP
NodePort:                 &lt;unset&gt;  32470/TCP
Endpoints:                192.168.212.125:8100
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;

[root@k8s-master ~]# kubectl describe service currency-exchange
Name:                     currency-exchange
Namespace:                default
Labels:                   app=currency-exchange
Annotations:              &lt;none&gt;
Selector:                 app=currency-exchange
Type:                     NodePort
IP Families:              &lt;none&gt;
IP:                       10.110.232.189
IPs:                      10.110.232.189
Port:                     &lt;unset&gt;  8000/TCP
TargetPort:               8000/TCP
NodePort:                 &lt;unset&gt;  31776/TCP
Endpoints:                192.168.19.160:8000,192.168.212.65:8000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>
<p>I just discovered wrong behavior in all coredns PODS, a lot of timeouts:</p>
<pre><code>[root@k8s-master ~]# kubectl logs coredns-74ff55c5b-zkkn7 -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:59744-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:53400-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:58465-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:58197-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:57794-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:43345-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:57361-&gt;192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:51716-&gt;192.168.100.1:53: i/o timeout
</code></pre>
<p>how can i start trace the problem?</p>
<p><strong>Extra Details:</strong></p>
<pre><code>[root@k8s-master ~]# kubectl exec -i -t currency-conversion-86d9bc4698-rxdkh -- sh
/ # wget http://currency-exchange:8000/currency-exchange/from/EUR/to/INR
wget: bad address 'currency-exchange:8000'
</code></pre>

<p>It looks to me that you have incorrectly setup your CNI overlay network. I checked your previous question to verify node's ip address and to me it looks that your pod network overlap with your host network:</p>
<p>The Kubernetes  <code>pod-network-cidr</code>  is the <code>IP prefix</code> for all pods in the Kubernetes cluster. This range must not clash with other networks in your <code>VPC</code></p>
<p>The Kubernetes pod network <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network" rel="nofollow noreferrer">documentation</a> describes this as well:</p>
<blockquote>
<p>Take care that your Pod network must not overlap with any of the host
networks: you are likely to see problems if there is any overlap. (If
you find a collision between your network plugin's preferred Pod
network and some of your host networks, you should think of a suitable
CIDR block to use instead, then use that during  <code>kubeadm init</code>  with
<code>--pod-network-cidr</code>  and as a replacement in your network plugin's
YAML).</p>
</blockquote>
<p>This is also mentioned in calico instructions when creating a <a href="https://docs.projectcalico.org/getting-started/kubernetes/quickstart#create-a-single-host-kubernetes-cluster" rel="nofollow noreferrer">cluster</a>:</p>
<blockquote>
<p><strong>Note</strong>: If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR, replacing 192.168.0.0/16 in
the above command.</p>
</blockquote>
<p>PS. You can always <code>wget</code> curl from <a href="https://github.com/moparisthebest/static-curl" rel="nofollow noreferrer">here</a>.</p>
