<p>I created a 3-node kubernetes cluster (1 master + 2 workers) on VirtualBox using the instructions <a href="http://shoddykid.blogspot.in/2018/05/kubernetes-on-virtualbox-cluster.html" rel="nofollow noreferrer">here</a>. I am using Flannel for the overlay network.</p>

<p>I set <code>sysctl -w net.bridge.bridge-nf-call-iptables=1</code> and <code>sysctl -w net.bridge.bridge-nf-call-ip6tables=1</code> on the master during installation. I hadn't set them on the workers at that time, but I set them later and rebooted both nodes.</p>

<ol>
<li><p>I have a trivial web app written in Go, listening on port 8080. I created a pod + replication controller thus:</p>

<pre><code>kubectl run foo --image=&lt;...&gt; --port=8080 --generator=run/v1
</code></pre>

<p>I am able to access my service using the POD IP and port 8080.</p></li>
<li><p>I also created a ClusterIP service.</p>

<pre><code>kubectl expose rc foo --name=foo-http --port=8081 --target-port=8080                   # ClusterIP service

# kubectl get svc foo-http
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
foo-http   ClusterIP   10.106.88.24   &lt;none&gt;        8081/TCP   14m
</code></pre>

<p>When I run this from any cluster node, it hangs:</p>

<pre><code>curl http://10.106.88.24:8081   # that's the ClusterIP
</code></pre>

<p>By running strace, I can see that curl initiates a non-blocking <code>connect</code>, and spins in a loop on <code>poll</code>, without the socket ever becoming ready for a <code>read</code> - so the connection doesn't go through.</p></li>
<li><p>If I create a NodePort service instead, I simply get connection refused.</p>

<pre><code># cat svc_nodeport.json
apiVersion: v1
kind: Service
metadata:
  name: foo-http
spec:
  type: NodePort
  ports:
  - port: 8081
    targetPort: 8080
    nodePort: 31123
  selector:
    app: foo

# kubectl create -f svc_nodeport.json
# kubectl get svc foo-http
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
foo-http   NodePort    10.104.78.88   &lt;none&gt;        8081:31123/TCP   7m
</code></pre>

<p>When I try connecting via port 31123:</p>

<pre><code># curl http://&lt;node-ip&gt;:31123    # Tried on master and both workers
curl: (7) Failed connect to &lt;node-ip&gt;:31123; Connection refused
</code></pre></li>
</ol>

<p>How do debug this?</p>

CppNoob
<p>I think your service manifest selector isn't selecting the pod.
Try to describe the pod and check for the labels.
Kubectl describe pod 
Then change your service manifest according to your pod labels.</p>

<p>Best practice is to use declarative ways by writing the deployment or pod manifest, instead of using the imperative commands of kubectl, such as run or create.</p>
