<p>So I've got a Kubernetes cluster up and running using the <a href="https://coreos.com/kubernetes/docs/latest/getting-started.html" rel="nofollow noreferrer">Kubernetes on CoreOS Manual Installation Guide</a>.</p>
<pre><code>$ kubectl get no
NAME              STATUS                     AGE
coreos-master-1   Ready,SchedulingDisabled   1h
coreos-worker-1   Ready                      54m

$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {&quot;health&quot;: &quot;true&quot;}
etcd-2               Healthy   {&quot;health&quot;: &quot;true&quot;}
etcd-1               Healthy   {&quot;health&quot;: &quot;true&quot;}

$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY     STATUS    RESTARTS   AGE       IP               NODE
default       curl-2421989462-h0dr7                     1/1       Running   1          53m       10.2.26.4        coreos-worker-1
kube-system   busybox                                   1/1       Running   0          55m       10.2.26.3        coreos-worker-1
kube-system   kube-apiserver-coreos-master-1            1/1       Running   0          1h        192.168.0.200   coreos-master-1
kube-system   kube-controller-manager-coreos-master-1   1/1       Running   0          1h        192.168.0.200   coreos-master-1
kube-system   kube-proxy-coreos-master-1                1/1       Running   0          1h        192.168.0.200   coreos-master-1
kube-system   kube-proxy-coreos-worker-1                1/1       Running   0          58m       192.168.0.204   coreos-worker-1
kube-system   kube-scheduler-coreos-master-1            1/1       Running   0          1h        192.168.0.200   coreos-master-1

$ kubectl get svc --all-namespaces
NAMESPACE   NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     kubernetes   10.3.0.1     &lt;none&gt;        443/TCP   1h
</code></pre>
<p>As with the guide, I've setup a service network <code>10.3.0.0/16</code> and a pod network <code>10.2.0.0/16</code>. Pod network seems fine as busybox and curl containers get IPs. But the services network has problems. Originally, I've encountered this when deploying <code>kube-dns</code>: the service IP <code>10.3.0.1</code> couldn't be reached, so kube-dns couldn't start all containers and DNS was ultimately not working.</p>
<p>From within the curl pod, I can reproduce the issue:</p>
<pre><code>[ root@curl-2421989462-h0dr7:/ ]$ curl https://10.3.0.1
curl: (7) Failed to connect to 10.3.0.1 port 443: No route to host

[ root@curl-2421989462-h0dr7:/ ]$ ip route
default via 10.2.26.1 dev eth0
10.2.0.0/16 via 10.2.26.1 dev eth0
10.2.26.0/24 dev eth0  src 10.2.26.4
</code></pre>
<p>It seems ok that there's only a default route in the container. As I understood it, the request (to default route) should be intercepted by the  <code>kube-proxy</code> on the worker node, forwarded to the the proxy on the master node where the IP is translated via iptables to the masters public IP.</p>
<p>There seems to be a common problem with a bridge/netfilter sysctl setting, but that seems fine in my setup:</p>
<pre><code>core@coreos-worker-1 ~ $ sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1
</code></pre>
<p>I'm having a real hard time to troubleshoot, as I lack the understanding of what the service IP is used for, how the service network is supposed to work in terms of traffic flow and how to best debug this.</p>
<p>So here're the questions I have:</p>
<ul>
<li>What is the 1st IP of the service network (10.3.0.1 in this case) used for?</li>
<li>Is above description of the traffic flow correct? If not, what steps does it take for a container to reach a service IP?</li>
<li>What are the best ways to debug each step in the traffic flow? (I can't get any idea what's wrong from the logs)</li>
</ul>
<p>Thanks!</p>

<p>The Sevice network provides fixed IPs for Services. It is not a routeable network (so don't expect <code>ip ro</code> to show anything nor will ping work) but a collection iptables rules managed by kube-proxy on each node (see <code>iptables -L; iptables -t nat -L</code> on the nodes, not Pods). These <a href="https://kubernetes.io/docs/user-guide/services/#virtual-ips-and-service-proxies" rel="noreferrer">virtual IPs</a> (see the pics!) act as load balancing proxy for endpoints (<code>kubectl get ep</code>), which are usually ports of Pods (but not always) with a specific set of labels as defined in the Service.</p>

<p>The first IP on the Service network is for reaching the kube-apiserver itself. It's listening on port 443 (<code>kubectl describe svc kubernetes</code>).</p>

<p>Troubleshooting is different on each network/cluster setup. I would generally check:</p>

<ul>
<li>Is kube-proxy running on each node? On some setups it's run via systemd and on others there is a DeamonSet that schedules a Pod on each node. On your setup it is deployed as static Pods created by the kubelets thrmselves from <code>/etc/kubernetes/manifests/kube-proxy.yaml</code></li>
<li>Locate logs for kube-proxy and find clues (can you post some?)</li>
<li>Change kube-proxy into <code>userspace</code> mode. Again, the details depend on your setup. For you it's in the file I mentioned above. Append <code>--proxy-mode=userspace</code> as a parameter <strong>on each node</strong></li>
<li>Is the overlay (pod) network functional?</li>
</ul>

<p>If you leave comments I will get back to you..</p>
