<p>I'm relatively new (&lt; 1 year) to GCP, and I'm still in the process of mapping the various services onto my existing networking mental model.</p>

<p>Once knowledge gap I'm struggling to fill is how HTTP requests are load balanced to services running in our GKE clusters.</p>

<p>On a test cluster, I created a service in front of pods that serve HTTP:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: contour
spec:
 ports:
 - port: 80
   name: http
   protocol: TCP
   targetPort: 8080
 - port: 443
   name: https
   protocol: TCP
   targetPort: 8443
 selector:
   app: contour
 type: LoadBalancer
</code></pre>

<p>The service is listening on node ports 30472 and 30816.:</p>

<pre><code>$ kubectl get svc contour
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
contour   LoadBalancer   10.63.241.69   35.x.y.z   80:30472/TCP,443:30816/TCP   41m
</code></pre>

<p>A GCP network load balancer is automatically created for me. It has its own public IP at 35.x.y.z, and is listening on ports 80-443:</p>

<p><a href="https://i.stack.imgur.com/VAimU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VAimU.png" alt="auto load balancer"></a></p>

<p>Curling the load balancer IP works:</p>

<pre><code>$ curl -q -v 35.x.y.z
* TCP_NODELAY set
* Connected to 35.x.y.z (35.x.y.z) port 80 (#0)
&gt; GET / HTTP/1.1
&gt; Host: 35.x.y.z
&gt; User-Agent: curl/7.62.0
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 404 Not Found
&lt; date: Mon, 07 Jan 2019 05:33:44 GMT
&lt; server: envoy
&lt; content-length: 0
&lt;
</code></pre>

<p>If I ssh into the GKE node, I can see the <code>kube-proxy</code> is listening on the service nodePorts (30472 and 30816) and nothing has a socket listening on ports 80 or 443:</p>

<pre><code># netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:20256         0.0.0.0:*               LISTEN      1022/node-problem-d 
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1221/kubelet        
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1369/kube-proxy     
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      297/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      330/sshd            
tcp6       0      0 :::30816                :::*                    LISTEN      1369/kube-proxy     
tcp6       0      0 :::4194                 :::*                    LISTEN      1221/kubelet        
tcp6       0      0 :::30472                :::*                    LISTEN      1369/kube-proxy     
tcp6       0      0 :::10250                :::*                    LISTEN      1221/kubelet        
tcp6       0      0 :::5355                 :::*                    LISTEN      297/systemd-resolve 
tcp6       0      0 :::10255                :::*                    LISTEN      1221/kubelet        
tcp6       0      0 :::10256                :::*                    LISTEN      1369/kube-proxy
</code></pre>

<p>Two questions:</p>

<ol>
<li>Given nothing on the node is listening on ports 80 or 443, is the load balancer directing traffic to ports 30472 and 30816?</li>
<li>If the load balancer is accepting traffic on 80/443 and forwarding to 30472/30816, where can I see that configuration? Clicking around the load balancer screens I can't see any mention of ports 30472 and 30816.</li>
</ol>

James Healy
<p>I think I found the answer to my own question - can anyone confirm I'm on the right track?</p>

<p>The network load balancer redirects the traffic to a node in the cluster without modifying the packet - packets for port 80/443 still have port 80/443 when they reach the node.</p>

<p>There's nothing listening on ports 80/443 on the nodes. However <code>kube-proxy</code> has written iptables rules that match packets <strong>to</strong> the load balancer IP, and rewrite them with the appropriate ClusterIP and port:</p>

<p>You can see the iptables config on the node:</p>

<pre><code>$ iptables-save | grep KUBE-SERVICES | grep loadbalancer                                                                                                            
-A KUBE-SERVICES -d 35.x.y.z/32 -p tcp -m comment --comment "default/contour:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-D53V3CDHSZT2BLQV                                                                 
-A KUBE-SERVICES -d 35.x.y.z/32 -p tcp -m comment --comment "default/contour:https loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-J3VGAQUVMYYL5VK6  

$ iptables-save | grep KUBE-SEP-ZAA234GWNBHH7FD4
:KUBE-SEP-ZAA234GWNBHH7FD4 - [0:0]
-A KUBE-SEP-ZAA234GWNBHH7FD4 -s 10.60.0.30/32 -m comment --comment "default/contour:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZAA234GWNBHH7FD4 -p tcp -m comment --comment "default/contour:http" -m tcp -j DNAT --to-destination 10.60.0.30:8080

$ iptables-save | grep KUBE-SEP-CXQOVJCC5AE7U6UC
:KUBE-SEP-CXQOVJCC5AE7U6UC - [0:0]
-A KUBE-SEP-CXQOVJCC5AE7U6UC -s 10.60.0.30/32 -m comment --comment "default/contour:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-CXQOVJCC5AE7U6UC -p tcp -m comment --comment "default/contour:https" -m tcp -j DNAT --to-destination 10.60.0.30:8443
</code></pre>

<p>An interesting implication is the the nodePort is created but doesn't appear to be used. That matches this comment in the <a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer" rel="nofollow noreferrer">kube docs</a>:</p>

<blockquote>
  <p>Google Compute Engine does not need to allocate a NodePort to make LoadBalancer work</p>
</blockquote>

<p>It also explains why GKE creates an automatic firewall rule that allows traffic from 0.0.0.0/0 towards ports 80/443 on the nodes. The load balancer isn't rewriting the packets, so the firewall needs to allow traffic from anywhere to reach iptables on the node, and it's rewritten there.</p>
