<p>I'm setting up an on-premise kubernetes cluster with kubeadm.</p>
<p>Here is the Kubernestes version</p>
<pre><code>clientVersion:
  buildDate: &quot;2022-10-12T10:57:26Z&quot;
  compiler: gc
  gitCommit: 434bfd82814af038ad94d62ebe59b133fcb50506
  gitTreeState: clean
  gitVersion: v1.25.3
  goVersion: go1.19.2
  major: &quot;1&quot;
  minor: &quot;25&quot;
  platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: &quot;2022-10-12T10:49:09Z&quot;
  compiler: gc
  gitCommit: 434bfd82814af038ad94d62ebe59b133fcb50506
  gitTreeState: clean
  gitVersion: v1.25.3
  goVersion: go1.19.2
  major: &quot;1&quot;
  minor: &quot;25&quot;
  platform: linux/amd64
</code></pre>
<p>I have installed metallb version 0.13.7</p>
<pre><code>kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml
</code></pre>
<p>Everything is running</p>
<pre><code>$ kubectl get all -n metallb-system
 
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-84d6d4db45-l2r55   1/1     Running   0          35s
pod/speaker-48qn4                 1/1     Running   0          35s
pod/speaker-ds8hh                 1/1     Running   0          35s
pod/speaker-pfbcp                 1/1     Running   0          35s
pod/speaker-st7n2                 1/1     Running   0          35s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/webhook-service   ClusterIP   10.104.14.119   &lt;none&gt;        443/TCP   35s

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   4         4         4       4            4           kubernetes.io/os=linux   35s

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           35s

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-84d6d4db45   1         1         1       35s
</code></pre>
<p>But when i try to apply an IPaddressPool CRD i get an error</p>
<pre><code>kubectl apply -f ipaddresspool.yaml
</code></pre>
<p>ipaddresspool.yaml file content</p>
<pre><code>apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.2.100-192.168.2.199
</code></pre>
<p>The error is a fail to call the validation webhook no route to host</p>
<pre><code>Error from server (InternalError): error when creating &quot;ipaddresspool.yaml&quot;: Internal error occurred: failed calling webhook &quot;ipaddresspoolvalidationwebhook.metallb.io&quot;: failed to call webhook: Post &quot;https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s&quot;: dial tcp 10.104.14.119:443: connect: no route to host
</code></pre>
<p>Here is the same error with line brakes</p>
<pre><code>Error from server (InternalError): 
error when creating &quot;ipaddresspool.yaml&quot;: 
Internal error occurred: failed calling webhook &quot;ipaddresspoolvalidationwebhook.metallb.io&quot;: 
failed to call webhook: 
Post &quot;https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s&quot;: 
dial tcp 10.104.14.119:443: connect: no route to host
</code></pre>
<p>The IP -address is correct</p>
<pre><code>NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
webhook-service   ClusterIP   10.104.14.119   &lt;none&gt;        443/TCP   18m
</code></pre>
<p>I have also tried installing metallb v 0.13.7 using helm but with the same result</p>
<p>Does someone know why the webhook cannot be called?</p>
<p><strong>EDIT</strong></p>
<p>As an answer to Thomas question, here is the description for webhook-service. NOTE that this is from <strong>another cluster</strong> with the <strong>same problem</strong> because I deleted the last cluster so the IP is not the same as last time</p>
<pre><code>$ kubectl describe svc webhook-service -n metallb-system

Name:              webhook-service
Namespace:         metallb-system
Labels:            &lt;none&gt;
Annotations:       &lt;none&gt;
Selector:          component=controller
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.105.157.72
IPs:               10.105.157.72
Port:              &lt;unset&gt;  443/TCP
TargetPort:        9443/TCP
Endpoints:         172.17.0.3:9443
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>Once understood the issue is fairly simple.</p>
<p>The metallb setup described above works as it is supposed to.
However, the Kubernetes setup does not. Most likely due to bad network configuration.</p>
<hr />
<h3>Understanding the error</h3>
<p>The key to understanding what is going on is the following error:</p>
<pre><code>Error from server (InternalError): error when creating &quot;ipaddresspool.yaml&quot;: Internal error occurred: failed calling webhook &quot;ipaddresspoolvalidationwebhook.metallb.io&quot;: failed to call webhook: Post &quot;https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s&quot;: dial tcp 10.104.14.119:443: connect: no route to host
</code></pre>
<p>Part of the applied metallb manifest is going to deploy a so-called <code>ValidatingWebhookConfiguration</code>.</p>
<p><a href="https://i.stack.imgur.com/gWdSM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gWdSM.png" alt="enter image description here" /></a></p>
<p>In the case of metallb this validating webhook will force the kube-apiserver to:</p>
<ol>
<li>send metallb-related objects like <code>IPAddressPool</code> to the webhook whenever someone creates or updates such an object</li>
<li>wait for the webhook to perform some checks on the object (e.g. validate that CIDRs and IPs are valid and not something like <code>481.9.141.12.27</code>)</li>
<li>and finally receive an answer from the webhook whether or not that object satisfies metallb's requirements and is allowed to be created (persisted to etcd)</li>
</ol>
<p>The error above pretty clearly suggests that the first out of the three outlined steps is failing.</p>
<hr />
<h3>Debugging</h3>
<p>To fix this error one has to debug the current setup, particularly the connection from the kube-apiserver to <code>webhook-service.metallb-system.svc:443</code>.</p>
<p>There is a wide range of possible network misconfigurations that could lead to the error. However, with the information available to us it is most likely going to be an error with the configured CNI.</p>
<p>Knowing that here is some help and a bit of guidance regarding the further debugging process:</p>
<p>Since the kube-apiserver is hardened by default it won't be possible to execute a shell into it.
For that reason one should deploy a debug application with the same network configuration as the kube-apiserver onto one of the control-plane nodes.
This can be achieved by executing the following command:</p>
<pre><code>kubectl debug -n kube-system node/&lt;control-plane-node&gt; -it --image=nicolaka/netshoot
</code></pre>
<p>Using common tools one can now reproduce the error inside the interactive shell. The following command is expected to fail (in a similar fashion to the kube-apiserver):</p>
<pre><code>curl -m 10 -k https://&lt;webhook-service-ip&gt;:443/
</code></pre>
<p>Given above error message it should fail due to bad routing on the node.
To check the routing table execute the following command:</p>
<pre><code>routel
</code></pre>
<blockquote>
<p>Does someone know why the webhook cannot be called?</p>
</blockquote>
<p>The output should show multiple CIDR ranges configured one of which is supposed to include the IP queried earlier.
Most likely the CIDR range in question will either be missing or a bad gateway configured which leads to the <code>no route to host</code> error.
It is the CNIs job to update routing tables on all nodes and ensure that nodes can reach these addresses so adding or editing new Kubernetes related entries to the routing table manually is not recommended.
Further debugging is dependent on the exact setup.
Depending on the setup and CNI of choice kube-proxy may or may not be involved in the issue as well.
However, inspecting the CNI configuration and logs is a good next step.</p>
<hr />
<h3>Some bonus information</h3>
<p>Some CNIs require the user to pay more attention to certain features and configuration as there can be issues involved otherwise.
Here are some popular CNIs that fall into this category:</p>
<ul>
<li>Calico (see <a href="https://metallb.universe.tf/configuration/calico/" rel="nofollow noreferrer">here</a>)</li>
<li>Weave (see <a href="https://metallb.universe.tf/configuration/weave/" rel="nofollow noreferrer">here</a>)</li>
<li>Kube-Router (see <a href="https://metallb.universe.tf/configuration/kube-router/" rel="nofollow noreferrer">here</a>)</li>
</ul>
