<p>I understand that, in Cloud scenarios, a LoadBalancer resource refers to and provisions an external layer 4 load balancer. There is some proprietary integration by which the cluster configures this load balancing. OK.</p>
<p>On-prem we have no such integration so we create our own load balancer outside of the cluster which distributes traffic to all nodes - effectively, in our case to the ingress.</p>
<p>I see no need for the cluster to know anything about this external load balancer.</p>
<p>What is a LoadBalancer* then in an on-prem scenario? Why do I have them? Why do I need one? What does it do? What happens when I create one? What role does the LoadBalancer resource play in ingress traffic. What effect does the LoadBalancer have outside the cluster? Is a LoadBalancer just a way to get a new IP address? How/why/ where does the IP point to and where does the IP come from?</p>
<ul>
<li>All questions refer to the Service of type “LoadBalancer” inside cluster and not my load balancer outside the cluster of which the cluster has no knowledge.</li>
</ul>

<p>As pointed out in the comments a kubernetes service of type LoadBalancer can not be used by default with on-prem setups. You can use <a href="https://metallb.universe.tf/" rel="nofollow noreferrer">metallb</a> to setup a service of that type in an on prem environment.</p>
<blockquote>
<p>Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. [...] If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created. [...] MetalLB aims to redress this imbalance by offering a network load balancer implementation that integrates with standard network equipment, so that external services on bare-metal clusters also “just work” as much as possible.</p>
</blockquote>
<p>You can for example use the BGP mode to advertise the service's IP to your router, read more on that in the <a href="https://metallb.universe.tf/concepts/bgp/" rel="nofollow noreferrer">docs</a>.</p>
<p>The project is still in beta but is promoted as production ready and used by several bigger companies.</p>
<p><strong>Edit</strong></p>
<p>Regarding your question in the comments:</p>
<blockquote>
<p>Can I just broadcast the MAC address of my node and manually add the IP I am broadcasting to the LoadBalancer service via kubectl edit?</p>
</blockquote>
<p>Yes that would work too. That's basically what metallb does, announcing the IP and updating the service.</p>
<p>Why need a software then? Imaging having 500 hosts that come and go with thousends of services of type <code>LoadBalancer</code> that come and go. You need an automation here.</p>
<blockquote>
<p>Why does Kubernetes need to know this IP?</p>
</blockquote>
<p>It doesn't. If you don't use an external ip, the service is still usable via it's NodePort, see for example the <a href="https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports" rel="nofollow noreferrer">istio docs</a> (with a little more details added by me):</p>
<pre><code>$ kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)
istio-ingressgateway   LoadBalancer   192.12.129.119   &lt;Pending&gt;   [...],80:32123/TCP,443:30994/TCP,[...]
</code></pre>
<p>Here the external IP is not set and stays in <code>&lt;Pending&gt;</code>. You can still use the service by pointing your traffic to <code>&lt;Node-IP&gt;:32123</code> for plain http and to <code>&lt;Node-IP&gt;:30994</code> for https. As you can see above those ports are mapped to 80 and 443.</p>
<p>If the external ip is set you can direct traffic directly to port 80 and 443 on the external load balancer. Kube-Proxy will create an iptables chain with the destination of you external ip, that basically leads from the external IP over the service ip with a load balancer configuration to a pod ip.</p>
<p>To investigate that set up a service of type LoadBalancer, make sure it has an external ip, connect to the host and run the iptables-save command (e.g. <code>iptables-save | less</code>). Search for the external ip and follow the chain until you end up at the pod.</p>
