<p>I want to have a tcpdump of a node port (like 30034) of a NodePort service pointing a pod in Kubernetes cluster.
This node port service is mapped inside an ingress resource under paths. When I hit the ingress using the host configured inside ingress, I get a response from the target pod but tcpdump doesn't trace anything. (Ingress-->NodePortService-> NodePort--[tcpdump]->pod)</p>

<p>I have tried with: sudo tcpdump -i any port 30034 -w tcp-dump.pcap
but its not capturing anything.</p>

<p>Could you please suggest here. What is the reason that tcpdump is not capturing anything when traffic comes via ingress controller.
However, if I hit the node directly as <a href="https://node-ip:30034:/service" rel="nofollow noreferrer">https://node-ip:30034:/service</a>; I receive the tcpdump.</p>

<p>Thanks.</p>

Jaraws
<p>TCPdump effectively in Kubernetes is a bit tricky and requires you to create a side car to your pod. What you are facing is actually the expected behavior. </p>

<blockquote>
  <p>run good old stuff like TCPdump or ngrep would not yield much
  interesting information, because you link directly to the bridge
  network or overlay in a default scenario.</p>
  
  <p>The good news is, that you can link your TCPdump container to the host
  network or even better, to the container network stack.
  Source: <a href="https://medium.com/@xxradar/how-to-tcpdump-effectively-in-docker-2ed0a09b5406" rel="nofollow noreferrer">How to TCPdump effectively in Docker</a></p>
</blockquote>

<p>The thing is that you have two entry points, one is for nodeIP:NodePort the second is ClusterIP:Port. Both are pointing to the same set of randomization rules for endpoints set on kubernetes iptables. </p>

<p>As soon as it can happen on any node it's hard to configure tcpdump to catch all interesting traffic in just one point.</p>

<p>The best tool I know for such kind of analysis is Istio, but it works mostly for HTTP traffic.</p>

<p>Considering this, the best solution is to use a tcpdumper sidecar for each pod behind the service.</p>

<p>Let's go trough an example on how to achieve this </p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web-app
        image: nginx
        imagePullPolicy: Always        
        ports:
        - containerPort: 80
          protocol: TCP
      - name: tcpdumper
        image: docker.io/dockersec/tcpdump
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: web-svc
  namespace: default
spec:
  ports:
  - nodePort: 30002
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: web
  type: NodePort
</code></pre>

<p>On this manifest we can notice tree important things. We have a nginx container and one tcpdumper container as a side car and we have a service defined as NodePort. </p>

<p>To access our sidecar, you have to run the following command: </p>

<pre><code>$ kubectl attach -it web-app-db7f7c59-d4xm6 -c tcpdumper
</code></pre>

<p>Example: </p>

<pre><code>$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        &lt;none&gt;        443/TCP        13d
web-svc      NodePort    10.108.142.180   &lt;none&gt;        80:30002/TCP   9d
</code></pre>

<pre><code>$ curl localhost:30002
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
&lt;style&gt;
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
&lt;/style&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
&lt;p&gt;If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.&lt;/p&gt;

&lt;p&gt;For online documentation and support please refer to
&lt;a href="http://nginx.org/"&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
Commercial support is available at
&lt;a href="http://nginx.com/"&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<pre><code>$ kubectl attach -it web-app-db7f7c59-d4xm6 -c tcpdumper
Unable to use a TTY - container tcpdumper did not allocate one
If you don't see a command prompt, try pressing enter.
&gt; web-app-db7f7c59-d4xm6.80: Flags [P.], seq 1:78, ack 1, win 222, options [nop,nop,TS val 300957902 ecr 300958061], length 77: HTTP: GET / HTTP/1.1
12:03:16.884512 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.1336: Flags [.], ack 78, win 217, options [nop,nop,TS val 300958061 ecr 300957902], length 0
12:03:16.884651 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.1336: Flags [P.], seq 1:240, ack 78, win 217, options [nop,nop,TS val 300958061 ecr 300957902], length 239: HTTP: HTTP/1.1 200 OK
12:03:16.884705 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.1336: Flags [P.], seq 240:852, ack 78, win 217, options [nop,nop,TS val 300958061 ecr 300957902], length 612: HTTP
12:03:16.884743 IP 192.168.250.64.1336 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 240, win 231, options [nop,nop,TS val 300957902 ecr 300958061], length 0
12:03:16.884785 IP 192.168.250.64.1336 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 852, win 240, options [nop,nop,TS val 300957902 ecr 300958061], length 0
12:03:16.889312 IP 192.168.250.64.1336 &gt; web-app-db7f7c59-d4xm6.80: Flags [F.], seq 78, ack 852, win 240, options [nop,nop,TS val 300957903 ecr 300958061], length 0
12:03:16.889351 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.1336: Flags [F.], seq 852, ack 79, win 217, options [nop,nop,TS val 300958062 ecr 300957903], length 0
12:03:16.889535 IP 192.168.250.64.1336 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 853, win 240, options [nop,nop,TS val 300957903 ecr 300958062], length 0
12:08:10.336319 IP6 fe80::ecee:eeff:feee:eeee &gt; ff02::2: ICMP6, router solicitation, length 16
12:15:47.717966 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [S], seq 3314747302, win 28400, options [mss 1420,sackOK,TS val 301145611 ecr 0,nop,wscale 7], length 0
12:15:47.717993 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.2856: Flags [S.], seq 2539474977, ack 3314747303, win 27760, options [mss 1400,sackOK,TS val 301145769 ecr 301145611,nop,wscale 7], length 0
12:15:47.718162 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 1, win 222, options [nop,nop,TS val 301145611 ecr 301145769], length 0
12:15:47.718164 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [P.], seq 1:78, ack 1, win 222, options [nop,nop,TS val 301145611 ecr 301145769], length 77: HTTP: GET / HTTP/1.1
12:15:47.718191 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.2856: Flags [.], ack 78, win 217, options [nop,nop,TS val 301145769 ecr 301145611], length 0
12:15:47.718339 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.2856: Flags [P.], seq 1:240, ack 78, win 217, options [nop,nop,TS val 301145769 ecr 301145611], length 239: HTTP: HTTP/1.1 200 OK
12:15:47.718403 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.2856: Flags [P.], seq 240:852, ack 78, win 217, options [nop,nop,TS val 301145769 ecr 301145611], length 612: HTTP
12:15:47.718451 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 240, win 231, options [nop,nop,TS val 301145611 ecr 301145769], length 0
12:15:47.718489 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 852, win 240, options [nop,nop,TS val 301145611 ecr 301145769], length 0
12:15:47.723049 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [F.], seq 78, ack 852, win 240, options [nop,nop,TS val 301145612 ecr 301145769], length 0
12:15:47.723093 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.2856: Flags [F.], seq 852, ack 79, win 217, options [nop,nop,TS val 301145770 ecr 301145612], length 0
12:15:47.723243 IP 192.168.250.64.2856 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 853, win 240, options [nop,nop,TS val 301145612 ecr 301145770], length 0
12:15:50.493995 IP 192.168.250.64.31340 &gt; web-app-db7f7c59-d4xm6.80: Flags [S], seq 124258064, win 28400, options [mss 1420,sackOK,TS val 301146305 ecr 0,nop,wscale 7], length 0
12:15:50.494022 IP web-app-db7f7c59-d4xm6.80 &gt; 192.168.250.64.31340: Flags [S.], seq 3544403648, ack 124258065, win 27760, options [mss 1400,sackOK,TS val 301146463 ecr 301146305,nop,wscale 7], length 0
12:15:50.494189 IP 192.168.250.64.31340 &gt; web-app-db7f7c59-d4xm6.80: Flags [.], ack 1, win 222, options 
</code></pre>

<p>You can also take a look at <a href="https://github.com/eldadru/ksniff" rel="nofollow noreferrer">ksniff</a> tool, a kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster.</p>
