<p>I am going through <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way" rel="nofollow noreferrer">kubernetes the hard way</a> to learn more about how to use it.  I am implementing it on openstack vm's.  The pod networks local to each individual kubernetes worker need to be able to communicate with one another via static routes.  In order to set up static routes with IP addresses other than the primary IP address configured in openstack I have to use <code>allowed_address_pairs</code> in openstack.  This works if I try to route on eth0, but if I try to route on the CNI interfaces (cnio0) it fails.</p>

<p>I configured static routes as a test between secondary IP addresses configured as aliases to the worker nodes' primary interfaces.  I set ip forwarding to 1 in proc.  In openstack, I configured the <code>allowed_address_pairs</code> using the secondary IP addresses and their associated mac addresses.</p>

<p>This routing configuration worked.</p>

<p>I then enabled the cni and fired up a few pods which triggered kubelet to create a cnio0 interface on each host and apply the POD CIDR IPs to those interfaces.  I added these IPs and their respective mac addresses to the <code>allowed_address_pairs</code> as well, but these interfaces do not route.  I am not able to reach the IPs from each individual box.</p>

<p>I expected to be able to successfully ping from node 1's cni0 IP address to node 2's cni0 IP address.</p>

<p>I watched the communication between the hosts in tcpdump while pinging from node 1's cnio0 interface to node 2's cnio0 interface. On node 1 I can see traffic going out.  On node 2 I can see the traffic being received and sent back, but I never see the return packet on node 1.</p>

<p>This is the exact same behavior I saw on the aliased interfaces when I tested static routes between them without configuring the <code>allowed_address_pairs</code> in openstack.  </p>

<p>How can I get the cnio0 interfaces to work with <code>allowed_address_pairs</code> so they can route between one another?</p>

svenwinkle
<p>I just found that setting the mac_address of the allowed_address_pairs to the mac address of the primary interface (the one configured in openstack) instead of the cnio0 interface makes everything work.</p>
