<p>I am trying to configure a K8s cluster on-prem and the servers are running Fedora CoreOS using multiple NICs. </p>

<p>I am configuring the cluster to use a non-default NIC - a bond which is defined with 2 interfaces. All servers can reach each-other over that interface and have HTTP + HTTPS connectivity to the internet.</p>

<p>kubeadm join hangs at:</p>

<pre><code>I0513 13:24:55.516837   16428 token.go:215] [discovery] Failed to request cluster-info, will try again: Get https://${BOND_IP}:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
</code></pre>

<p>The relevant kubeadm init config looks like this:</p>

<pre><code>[...]
localAPIEndpoint:
  advertiseAddress: ${BOND_IP}
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
    runtime-cgroups: "/systemd/system.slice"
    kubelet-cgroups: "/systemd/system.slice"
    node-ip: ${BOND_IP}
  criSocket: /var/run/dockershim.sock
  name: master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
[...]
</code></pre>

<p>The join config that am using looks like this:</p>

<pre><code>apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: ${TOKEN}
    caCertHashes:
    - "${SHA}"
    apiServerEndpoint: "${BOND_IP}:6443"
nodeRegistration:
  kubeletExtraArgs:
    volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
    runtime-cgroups: "/systemd/system.slice"
    kubelet-cgroups: "/systemd/system.slice"
</code></pre>

<p>If I am trying to configure it using default eth0, it works without issues.</p>

<p>This is not a connectivity issue. The port test works fine:</p>

<pre><code># nc -s ${BOND_IP_OF_NODE} -zv ${BOND_IP_OF_MASTER} 6443
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to ${BOND_IP_OF_MASTER}:6443.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

</code></pre>

<p>I suspect this happens due to kubelet listening on eth0, if so, can I change it to use a different NIC/IP?</p>

<p>LE: The eth0 connection has been cut off completely (cable out, interface down, connection down).
Now, when we init, if we choose port 0.0.0.0 for the kube-api it defaults to the bond, which we wanted initially:</p>

<pre><code>kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 0.0.0.0
</code></pre>

<p>result:</p>

<pre><code>[certs] apiserver serving cert is signed for DNS names [emp-prod-nl-hilv-quortex19 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.0.0.1 ${BOND_IP}]
</code></pre>

<p>I have even added the 6443 port in iptables for accept and it still times out.. All my CALICO pods are up and running (all pods for that matter in kube-system namespace)</p>

<p>LLE:</p>

<p>I have tested calico and weavenet and both show the same issue. The api-server is up and can be reached from the master using curl but it times out from the nodes.</p>

<p>LLLE:</p>

<p>On the premise that the kube-api is nothing but an HTTPS server, I have tried two options from the node that cannot reach it when doing the kubeadm join:</p>

<ol>
<li>Ran a python3 simple http server over 6443 and WAS ABLE TO CONNECT from node</li>
<li>Ran an nginx pod and exposed it over another port as NodePort and WAS ABLE TO CONNECT from node</li>
</ol>

<p>the node just cant reach the api-server on 6443 or any other port for that matter .... </p>

<p>what am i doing wrong...</p>

Florin
<p>The cause:</p>

<p>The interface used was in BOND of type ACTIVE-ACTIVE. This made it so kubeadm tried another interface from the 2 bonded, which was not in the same subnet as the IP of the advertised server apparently...</p>

<p>Using ACTIVE-PASSIVE did the trick and was able to join the nodes.</p>

<p>LE: If anyone knows why <strong>kubeadm join</strong> does not support <strong>LACP</strong> with <strong>ACTIVE-ACTIVE bond</strong> setups on <strong>FEDORA COREOS</strong> please advise here. Otherwise, if additional configurations are required, I would very much like to know what I have missed.</p>
