title: How do you route return packets through same gateway doing the port forwarding?
body: <p><strong>DESCRIPTION</strong>
  I have HTTP and SMTP Linux servers and two Linux routers I created using firewalld on the same LAN.  Both routers act great as Internet gateways.  One works great to handle port forwarding of public incoming to the servers in the LAN.  I use a mobile device not on wifi, but cell only, to test outside traffic, loading a page.</p>
  <p>However, I'm trying to get the second router to be a redundant backup and point to the same servers using different public IPs.  The problem is each server only works with the port forwarding coming from its default gateway IP.  If I switch the HTTP servers gateway IP to the other router, it then only works with the other router's port forwarding, but never both at the same time, despite adding both gateways to the HTTP server's routing table with different metrics.</p>
  <p>To be clear, it always receives the packets from both routers, and it attempts to return packets to the public client IP (mobile IP when testing).</p>
  <p>I can see from TCPDUMP that it returns packets to the public IP that initiated it.  But, the clients, be it web browsers or SMTP clients, can never complete the session, such as loading the page.  I suspect it sees the packets coming from a different server with a different public IP, and does not associate them with the session, dropping them. I have to use mobile to test outside, so cannot TCPDUMP that.  But, the behavior when I switch the gateways is pretty strong evidence that's what happening.</p>
  <p><strong>QUESTION</strong>: How can I get it to return the packets out of the gateway where it received the port forwarding from instead of using its default gateway, or otherwise resolve this?</p>
  <p><em>NOTE</em>: If I allow masquerade on the internal zone of a router for all sources, the client loads the page without a problem.  But, THIS IS NOT AN OPTION as it is a huge security problem as the servers only see the IP of the routers, not the client public IPs, turning an SMTP server into an open relay and having no way to secure by IP.  This helps confirm the routing dilemma.  It understandably resolves it as it will always route back to the router that did the port forwarding since that becomes the destination IP regardless of which gateway it uses.</p>
  <p>This also works just fine internally via all public IPs, probably because I allow masquerading on the internal zones when the source is internal.</p>