<p>I'm trying to create 3 instances of Kafka and deploy it a local Kubernetes setup. Because each instance needs some specific configuration, I'm creating one RC and one service for each - eagerly waiting for <a href="https://github.com/kubernetes/kubernetes/pull/18016" rel="noreferrer">#18016</a> ;)</p>

<p>However, I'm having problems because Kafka can't establish a network connection to itself when it uses the service IP (a Kafka broker tries to do this when it is exchanging replication messages with other brokers). For example, let's say I have two worker hosts (172.17.8.201 and 172.17.8.202) and my pods are scheduled like this:</p>

<ul>
<li><p>Host 1 (172.17.8.201)</p>

<ul>
<li><code>kafka1</code> pod (10.2.16.1)</li>
</ul></li>
<li><p>Host 2 (172.17.8.202)</p>

<ul>
<li><code>kafka2</code> pod (10.2.68.1)</li>
<li><code>kafka3</code> pod (10.2.68.2)</li>
</ul></li>
</ul>

<p>In addition, let's say I have the following service IPs:</p>

<ul>
<li><code>kafka1</code> cluster IP: 11.1.2.96</li>
<li><code>kafka2</code> cluster IP: 11.1.2.120</li>
<li><code>kafka3</code> cluster IP: 11.1.2.123</li>
</ul>

<p>The problem happens when the <code>kafka1</code> pod (container) tries to send a message (to itself) using the <code>kafka1</code> cluster IP (11.1.2.96). For some reason, the connection cannot established and the message is not sent.</p>

<p>Some more information: If I manually connect to the <code>kafka1</code> pod, I can correctly telnet to <code>kafka2</code> and <code>kafka3</code> pods using their respective cluster IPs (11.1.2.120 / 11.1.2.123). Also, if I'm in the <code>kafka2</code> pod, I connect to both <code>kafka1</code> and <code>kafka3</code> pods using 11.1.2.96 and 11.1.2.123. Finally, I can connect to all pods (from all pods) if I use the pod IPs.</p>

<p>It is important to emphasize that I shouldn't tell the kafka brokers to use the pod IPs instead of the cluster IPs for replication. As it is right now, Kafka uses for replication whatever IP you configure to be "advertised" - which is the IP that your client uses to connect to the brokers. Even if I could, I believe this problem may appear with other software as well.</p>

<p>This problem seems to happen only with the combination I am using, because the exact same files work correctly in GCE. Right now, I'm running:</p>

<ul>
<li>Kubernetes 1.1.2</li>
<li>coreos 928.0.0</li>
<li>network setup with flannel</li>
<li>everything on vagrant + VirtualBpx</li>
</ul>

<p>After some debugging, I'm not sure if the problem is in the workers iptables rules, in kube-proxy, or in flannel. </p>

<p>PS: I posted this question originally as an <a href="https://github.com/kubernetes/kubernetes/issues/19930" rel="noreferrer">Issue</a> on their github, but I have been redirected to here by the Kubernetes team. I reword the text a bit because it was sounding like it was a "support request", but actually I believe it is some sort of bug. Anyway, sorry about that Kubernetes team! </p>

<hr>

<p><strong>Edit:</strong> This problem has been confirmed as a bug <a href="https://github.com/kubernetes/kubernetes/issues/20391" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/20391</a></p>

<p>for what you want to do you should be using a <strong>Headless Service</strong>
<a href="http://kubernetes.io/v1.0/docs/user-guide/services.html#headless-services" rel="noreferrer">http://kubernetes.io/v1.0/docs/user-guide/services.html#headless-services</a></p>

<p>this means setting </p>

<p><code>clusterIP: None</code></p>

<p>in your <strong>Service</strong></p>

<p>and that means there won't be an IP associated with the service but it will return all IPs of the Pods selected by the <code>selector</code></p>
