<p>I have microservices running within EKS 1.22.
Is there a possible way to check the microservice communication also pod-to-pod communication within the cluster through CLI?</p>

dev
<p>Everything out of the box should work properly (assuming you have <a href="https://aws.amazon.com/eks/" rel="nofollow noreferrer">AWS EKS</a>).</p>
<p>I think this article - <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/" rel="nofollow noreferrer">Debug Services</a> has very helpful hints.</p>
<p>Let's first check pod to pod communication using trival method - <code>ping</code> command. I created two NGINX deployments (one in default namespace, second one in namespace <code>test</code>):</p>
<pre><code>kubectl create deployment nginx --image=nginx
kubectl create deployment nginx --image=nginx -n test
</code></pre>
<p>Now I will check IP addresses of both of them:</p>
<pre><code>user@shell:~$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP
nginx-6799fc88d8-jxpj4   1/1     Running   0          3m13s   172.17.0.2
</code></pre>
<p>And also in <code>test</code> namespace:</p>
<pre><code>user@shell:~$ kubectl get pods -o wide -n test
NAME                     READY   STATUS    RESTARTS   AGE    IP
nginx-6799fc88d8-z2glk   1/1     Running   0          103s   172.17.0.4
</code></pre>
<p>Now I will <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/" rel="nofollow noreferrer">execute into one pod</a> and check connectivity to the second one:</p>
<pre><code>user@shell:~$ kubectl exec -it nginx-6799fc88d8-jxpj4 -- sh
# apt update
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
...
All packages are up to date.
# apt-get install inetutils-ping
Reading package lists... Done
...
Setting up inetutils-ping (2:2.0-1) ...
# ping 172.17.0.4
PING 172.17.0.4 (172.17.0.4): 56 data bytes
64 bytes from 172.17.0.4: icmp_seq=0 ttl=64 time=0.058 ms
64 bytes from 172.17.0.4: icmp_seq=1 ttl=64 time=0.123 ms
</code></pre>
<p>Okay, so the pod to pod connection is working. Keep in mind that container images are minimal, so you may install <code>ping</code> as I did in my example. Also, depending on your application you can use different methods for checking connectivity - I could use <code>curl</code> command as well and I will get the standard NGINX home page.</p>
<p>Now time to create <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">services</a> (I am assuming this is what you mean by microservice) and test connectivity. Service is an abstract mechanism for exposing pods on a network. So we can test connectivity either by getting list of endpoints - IP address of the pods associated with this service - <code>kubectl get endpoints my-service</code>, and then checking pod to pod connection like in previous example, or we can just <code>curl</code> service IP  address/hostname. For hostname between namespaces it's little bit different! Check below:</p>
<p>Let's create deployments with 3 replicas:</p>
<pre><code>kubectl create deployment nginx --image=nginx --replicas=3
kubectl create deployment nginx --image=nginx --replicas=3 -n test
</code></pre>
<p>For each deployment we will <a href="https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/#creating-a-service-for-an-application-running-in-five-pods" rel="nofollow noreferrer">create service using <code>kubectl expose</code></a>:</p>
<pre><code>kubectl expose deployment nginx --name=my-service --port=80
kubectl expose deployment nginx --name=my-service-test --port=80 -n test
</code></pre>
<p>Time to get IP addresses of the services:</p>
<pre><code>user@shell:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP   64d
my-service   ClusterIP   10.107.224.54   &lt;none&gt;        80/TCP    12m
</code></pre>
<p>And in <code>test</code> namespace:</p>
<pre><code>user@shell:~$ kubectl get svc -n test
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
my-service-test   ClusterIP   10.110.51.62   &lt;none&gt;        80/TCP    8s
</code></pre>
<p>I will exec into pod in default namespace and <code>curl</code> IP address of the <code>my-service-test</code> in second namespace:</p>
<pre><code>user@shell:~$ kubectl exec -it nginx-6799fc88d8-w5q8s -- sh
# curl 10.110.51.62
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
&lt;style&gt;
</code></pre>
<p>Okay, it's working... Let's try with hostname:</p>
<pre><code># curl my-service-test
curl: (6) Could not resolve host: my-service-test
</code></pre>
<p>Not working... why? Let's check <code>/etc/resolv.conf</code> file:</p>
<pre><code># cat resolv.conf
nameserver 10.96.0.10
search test.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>
<p>It's looking for <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#does-the-service-work-by-dns-name" rel="nofollow noreferrer">hostnames only in namespace where pod is located</a>.</p>
<p>So pod in the <code>test</code> namespace will have something like:</p>
<pre><code># cat resolv.conf
nameserver 10.96.0.10
search test.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>
<p>Let's try to curl <code>my-service-test.test.svc.cluster.local</code> from pod in default namespace:</p>
<pre><code># curl my-service-test.test.svc.cluster.local
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
</code></pre>
<p>It's working.</p>
<p>To sum up:</p>
<ul>
<li>communication between pods in cluster should work properly between all namespaces (assuming that you have proper <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/" rel="nofollow noreferrer">CNI plugin</a> installed, on AWS EKS you have)</li>
<li>pods containers are in most cases Linux containers so just use <a href="https://geekflare.com/linux-test-network-connectivity/" rel="nofollow noreferrer">Linux tools</a> to check connectivity (like <code>ping</code> or <code>curl</code>)</li>
<li>Services are an abstract mechanism for exposing pods on a network, you can connect to them for example using <code>curl</code> command</li>
<li>IP addresses are cluster wide, hostnames are namespace wide - if you want to connect to the resource from the other namespace you need to use fully-qualified name (it applies for services, pods...)</li>
</ul>
<p>Also check these articles:</p>
<ul>
<li><a href="https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/" rel="nofollow noreferrer">Access Services Running on Clusters | Kubernetes</a></li>
<li><a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">Service | Kubernetes</a></li>
<li><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/" rel="nofollow noreferrer">Debug Services | Kubernetes</a></li>
<li><a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer">DNS for Services and Pods | Kubernetes</a></li>
</ul>
