<p>I am trying to build a networking application in a Kubernetes cluster which processes packets between two servers and offers a security service on the traffic going across the application. Following points summarize how this application works:</p>

<ol>
<li><p>Server1 and Server2 connect to the network gateway application over
IPsec tunnels.</p></li>
<li><p>The packets sent by Server1 are encapsulated in the IPsec tunnel and
    are destined to Server2, that is, the destination IP address in the
    inner packet is Server2 IP address.</p></li>
<li><p>IPsec server runs in a Pod in the cluster. This has been defined as
    a Kubernetes service IPSEC.</p></li>
<li><p>I have created another Kubernetes service, called TELEMETRY, which
    works on the network packets. This service collects some telemetry
    data from the packets and optionally applies some policies on the
    packets.</p></li>
<li><p>The decrypted packets received on this IPSEC pod having destination
    IP address same as Server2 IP address, need to be forwarded to
    TELEMETRY service pod.</p></li>
<li><p>The TELEMETRY service pod processes the packet and is supposed to
    forward it back to IPSEC pod so that the packet can be forwarded to
    the actual destination Server2 over an IPsec tunnel.</p></li>
<li><p>Once this will work, I also plan to add couple of services after
        TELEMETRY, so that the TELEMETRY service pod will forward the packet
        to those services. The last service in chain will forward it back to
        the IPSEC pod.</p></li>
</ol>

<p>I am facing challenges in forwarding the IP packets from one pod to another. I want to forward packets that are not destined to any service or pod in the cluster from one pod to another pod. I explored Flannel, but it does not forward the packets on overlay if the destination pod is running on the same node. It doesn't appear to be good solution from performance perspective due to multiple packet copies between user and kernel space.</p>

<p>It would be really a great help if someone can suggest a good solution for forwarding the IP packets from one pod to another pod in my application.</p>

<p>Instead of building this from scratch I would suggest to use a service mesh such as <a href="https://istio.io" rel="nofollow noreferrer">istio</a> for this. Istio and many other service meshes have data plane proxy such as <a href="https://www.envoyproxy.io" rel="nofollow noreferrer">envoy</a> which intercepts all traffic coming out of one pod and going into another pod. Service meshes offers security features such as mTLS, TLS certificates rotation and telemetry and many more out of the box.</p>

<p>If the control plane offered by open source service meshes does not support your specific use case you can write a custom control plane while still using envoy as data plane proxy.</p>
