<p>I want to setup a kubernetes cluster in KVM VMs that uses a flat DHCP network for all its communications.</p>

<p>The VNICs of the VMs, which are going to be kube-master and kube-workers are bridged to use an existing physical DHCP network.</p>

<p>I want to know if it is possible to make kubernetes to use the same physical DHCP network both for PODs and kubernetes internalities. Does anybody know how to configure it?</p>

<p>I tried to configure in /etc/cni/net.d a brige networking, but it did not work. Do I have to try IPVLAN, or anything else?</p>

<p>Thanks</p>

jfranzoy
<p>I recommend reading Kubernetes documentation about <a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/" rel="nofollow noreferrer">Cluster Networking</a>.</p>

<p>You might use <a href="https://github.com/Intel-Corp/multus-cni" rel="nofollow noreferrer">Multus</a> (a Multi Network plugin). The CNI enables attaching multiple network interfaces to pods in Kubernetes.</p>

<blockquote>
  <p><a href="https://github.com/Intel-Corp/multus-cni" rel="nofollow noreferrer">Multus</a>  is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.</p>
  
  <p>Multus supports all  <a href="https://github.com/containernetworking/plugins" rel="nofollow noreferrer">reference plugins</a>  (eg.  <a href="https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel" rel="nofollow noreferrer">Flannel</a>,  <a href="https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp" rel="nofollow noreferrer">DHCP</a>,  <a href="https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan" rel="nofollow noreferrer">Macvlan</a>) that implement the CNI specification and 3rd party plugins (eg.  <a href="https://github.com/projectcalico/cni-plugin" rel="nofollow noreferrer">Calico</a>,  <a href="https://github.com/weaveworks/weave" rel="nofollow noreferrer">Weave</a>,  <a href="https://github.com/cilium/cilium" rel="nofollow noreferrer">Cilium</a>,  <a href="https://github.com/contiv/netplugin" rel="nofollow noreferrer">Contiv</a>). In addition to it, Multus supports  <a href="https://github.com/hustcat/sriov-cni" rel="nofollow noreferrer">SRIOV</a>,  <a href="https://github.com/Intel-Corp/sriov-cni" rel="nofollow noreferrer">DPDK</a>,  <a href="https://github.com/intel/vhost-user-net-plugin" rel="nofollow noreferrer">OVS-DPDK &amp; VPP</a>  workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.</p>
</blockquote>

<p>With it you can use <a href="https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp" rel="nofollow noreferrer">dhcp plugin</a> </p>

<blockquote>
  <p>With dhcp plugin the containers can get an IP allocated by a DHCP server already running on your network. This can be especially useful with plugin types such as macvlan. Because a DHCP lease must be periodically renewed for the duration of container lifetime, a separate daemon is required to be running. The same plugin binary can also be run in the daemon mode.</p>
</blockquote>

<p>There is also <a href="https://github.com/npflan/dhcp" rel="nofollow noreferrer">NPF DHCPd on Kubernetes</a></p>

<blockquote>
  <p>This deployment uses several kubernetes features, these are mostly driven by assumptions and limitations of the old isc-dhcpd. The key points are:</p>
  
  <ul>
  <li>Isc-dhcpd must know the address of itself and its failover peers at config time, this obviously conflicts with kubernetes pod addressing and self-healing features, isc-dhcpd does however allow DNS resolution of these values, so we use kubernetes  <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer">dns records for pods and services</a>  to overcome these</li>
  <li>Isc-dhcpd stores its dhcp leases in a file, but it does not create the file automatically on startup. To ensure this always works as intended we use an  <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="nofollow noreferrer">init container</a>, to touch the file before starting dhcpd itself</li>
  <li>While a pod can DNS query its own FQDN to find its own IP, it cannot query other pods IPs by their FQDN, this derailed the process for a while, but it's actually neatly solved by making a  <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services" rel="nofollow noreferrer">headless service</a>, which will expose its pods IP addresses via DNS.</li>
  <li>The deployment and services are actually split into two; primary and secondary. This is a bit of a workaround, because dhcpd requires the configs for each instance to be slightly different, so we accomplish this by having two of everything, the advantage of still doing a deployment is that we get a replication controller to spin up a replacement instance if one fails.</li>
  <li>The two deployments share most of their config, except from a few details, so we can have a kubernetes  <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configmap/" rel="nofollow noreferrer">configmap</a>  with the shared config, and then the few different details which kubernetes will mount inside the pod, on the same location for the primary and secondary.</li>
  <li>Since we have two deployments, we want to make sure they don't get deployed on the same physical node in the cluster, for this we use  <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity" rel="nofollow noreferrer">anti-affinity</a>  so kubernetes will prefer to place it on separate nodes.</li>
  <li>It would possibly be more clean to use the new  <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">statefulset</a>, so we could avoid the duplicate deployments, this is something to look into for NPF 2018. We do not have support for PersistentVolumeClaims, but only PersistentVolumes in our iSCSI SAN, which is maybe a bit hard to do with statefulsets atm.</li>
  </ul>
</blockquote>

<p><strong>Update</strong></p>

<p>I think this might be the best solution for you, <a href="https://github.com/kvaps/bridget" rel="nofollow noreferrer">Simple bridge network for kubernetes</a>.
You have the whole deployment in <code>bridget.yaml</code> which is creating a DeamonSet](<a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/</a>) so it will run on all (or some) nodes will run a copy of the pod.</p>

<blockquote>
  <p>bridget - it's short shell script, that helps you for organise simple bridge network for Kubernetes. There is no overlays, no politics. Just flat L2-network across all your hosts and pods.</p>
  
  <p>In addition bridget can automatically configure VLAN and bridge interfaces for that. See the picture:</p>
  
  <p><a href="https://github.com/kvaps/bridget/blob/master/images/scheme.svg" rel="nofollow noreferrer"><img src="https://github.com/kvaps/bridget/raw/master/images/scheme.svg?sanitize=true" alt=""></a></p>
  
  <p>bridget automatically retrieves node cidr from your pod-network, and configures cni for use it.</p>
</blockquote>
