<p>I'm trying to create a <code>Kubernetes</code> cluster for learning purposes. So, I created 3 virtual machines with <code>Vagrant</code> where the master has IP address of <code>172.17.8.101</code> and the other two are <code>172.17.8.102</code> and <code>172.17.8.103</code>.</p>

<p>It's clear that we need <code>Flannel</code> so that our containers in different machines can connect to each other without port mapping. And for <code>Flannel</code> to work, we need <code>Etcd</code>, because flannel uses this <code>Datastore</code> to put and get its data. </p>

<p>I installed <code>Etcd</code> on master node and put <code>Flannel</code> network address on it with command <code>etcdctl set /coreos.com/network/config '{"Network": "10.33.0.0/16"}'</code></p>

<p>To enable <code>ip masquerading</code> and also using the private network interface in the virtual machine, I added <code>--ip-masq --iface=enp0s8</code> to <code>FLANNEL_OPTIONS</code> in <code>/etc/sysconfig/flannel</code> file.</p>

<p>In order to make <code>Docker</code> use <code>Flannel</code> network, I added <code>--bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}'</code> to <code>OPTIONS</code> variable in <code>/etc/sysconfig/docker</code> file. Note that the values for <code>FLANNEL_SUBNET</code> and <code>FLANNEL_MTU</code> variables are the ones set by <code>Flannel</code> in <code>/run/flannel/subnet.env</code> file.</p>

<p>After all these settings, I installed <code>kubernetes-master</code> and <code>kubernetes-client</code> on the master node and <code>kubernetes-node</code> on all the nodes. For the final configurations, I changed <code>KUBE_SERVICE_ADDRESSES</code> value in <code>/etc/kubernetes/apiserver</code> file to <code>--service-cluster-ip-range=10.33.0.0/16</code>
 and <code>KUBELET_API_SERVER</code> value in <code>/etc/kubernetes/kubelet</code> file to <code>--api-servers=http://172.17.8.101:8080</code>.</p>

<p>This is the link to <a href="https://github.com/yubar45/k8s-tutorial" rel="nofollow noreferrer">k8s-tutorial project</a> repository with the complete files.</p>

<p>After all these efforts, all the services start successfully and work fine. It's clear that there are 3 nodes running when I use the command <code>kubectl get nodes</code>. I can successfully create a <code>nginx</code> pod with command <code>kubectl run nginx-pod --image=nginx --port=80 --labels="app=nginx"</code> and create a service with <code>kubectl expose pod nginx-pod --port=8000 --target-port=80 --name="service-pod"</code> command.</p>

<p>The command <code>kubectl describe service service-pod</code> outputs the following results:</p>

<pre><code>Name:           service-pod
Namespace:      default
Labels:         app=nginx
Selector:       app=nginx
Type:           ClusterIP
IP:             10.33.39.222
Port:           &lt;unset&gt; 8000/TCP
Endpoints:      10.33.72.2:80
Session Affinity:   None
No events.
</code></pre>

<p>The challenge is that when I try to connect to the created service with <code>curl 10.33.79.222:8000</code> I get <code>curl: (7) Failed connect to 10.33.72.2:8000; Connection refused</code> but if I try <code>curl 10.33.72.2:80</code> I get the default <code>nginx</code> page. Also, I can't ping to <code>10.33.79.222</code> and all the packets get lost. </p>

<p>Some suggested to stop and disable <code>Firewalld</code>, but it wasn't running at all on the nodes. As <code>Docker</code> changed <code>FORWARD</code> chain policy to <code>DROP</code> in <code>Iptables</code> after version 1.13 I changed it back to <code>ACCEPT</code> but it didn't help either. I eventually tried to change the <code>CIDR</code> and use different IP/subnets but no luck.</p>

<p>Does anybody know where am I going wrong or how to figure out what's the problem that I can't connect to the created service?</p>

<p>The only thing I can see that you have that is conflicting is the PodCidr with Cidr that you are using for the services.</p>

<p>The Flannel network: <code>'{"Network": "10.33.0.0/16"}'</code>. Then on the kube-apiserver <code>--service-cluster-ip-range=10.33.0.0/16</code>. That's the same range and it should be different so you have your kube-proxy setting up services for <code>10.33.0.0/16</code> and then you have your overlay thinking it needs to route to the pods running on <code>10.33.0.0/16</code>.  I would start by choosing a completely non-overlapping Cidrs for both your pods and services.</p>

<p>For example on my cluster (I'm using Calico) I have a podCidr of <code>192.168.0.0/16</code> and I have a service Cidr of <code>10.96.0.0/12</code></p>

<p>Note: you wouldn't be able to ping <code>10.33.79.222</code> since ICMP is not allowed in this case.</p>
