title: Kubernetes pods can't ping each other using ClusterIP
body: <p>I'm trying to ping the kube-dns service from a dnstools pod using the cluster IP assigned to the kube-dns service. The ping request times out. From the same dnstools pod, I tried to curl the kube-dns service using the exposed port,  but that timed out as well.</p>
  
  <p>Following is the output of <code>kubectl get pods --all-namespaces -o wide</code></p>
  
  <pre><code>NAMESPACE       NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE
  default         pod/busybox                               1/1     Running   62         2d14h   192.168.1.37   kubenode      &lt;none&gt;
  default         pod/dnstools                              1/1     Running   0          2d13h   192.168.1.45   kubenode      &lt;none&gt;
  default         pod/nginx-deploy-7c45b84548-ckqzb         1/1     Running   0          6d11h   192.168.1.5    kubenode      &lt;none&gt;
  default         pod/nginx-deploy-7c45b84548-vl4kh         1/1     Running   0          6d11h   192.168.1.4    kubenode      &lt;none&gt;
  dmi             pod/elastic-deploy-5d7c85b8c-btptq        1/1     Running   0          2d14h   192.168.1.39   kubenode      &lt;none&gt;
  kube-system     pod/calico-node-68lc7                     2/2     Running   0          6d11h   10.62.194.5    kubenode      &lt;none&gt;
  kube-system     pod/calico-node-9c2jz                     2/2     Running   0          6d12h   10.62.194.4    kubemaster    &lt;none&gt;
  kube-system     pod/coredns-5c98db65d4-5nprd              1/1     Running   0          6d12h   192.168.0.2    kubemaster    &lt;none&gt;
  kube-system     pod/coredns-5c98db65d4-5vw95              1/1     Running   0          6d12h   192.168.0.3    kubemaster    &lt;none&gt;
  kube-system     pod/etcd-kubemaster                       1/1     Running   0          6d12h   10.62.194.4    kubemaster    &lt;none&gt;
  kube-system     pod/kube-apiserver-kubemaster             1/1     Running   0          6d12h   10.62.194.4    kubemaster    &lt;none&gt;
  kube-system     pod/kube-controller-manager-kubemaster    1/1     Running   1          6d12h   10.62.194.4    kubemaster    &lt;none&gt;
  kube-system     pod/kube-proxy-9hcgv                      1/1     Running   0          6d11h   10.62.194.5    kubenode      &lt;none&gt;
  kube-system     pod/kube-proxy-bxw9s                      1/1     Running   0          6d12h   10.62.194.4    kubemaster    &lt;none&gt;
  kube-system     pod/kube-scheduler-kubemaster             1/1     Running   1          6d12h   10.62.194.4    kubemaster    &lt;none&gt;
  kube-system     pod/tiller-deploy-767d9b9584-5k95j        1/1     Running   0          3d9h    192.168.1.8    kubenode      &lt;none&gt;
  nginx-ingress   pod/nginx-ingress-66wts                   1/1     Running   0          5d17h   192.168.1.6    kubenode      &lt;none&gt;
  </code></pre>
  
  <p>In the above output, why do some pods have an IP assigned in the <code>192.168.0.0/24</code> subnet whereas others have an IP that is equal to the IP address of my node/master? (10.62.194.4 is the IP of my master, 10.62.194.5 is the IP of my node)</p>
  
  <p>This is the <code>config.yml</code> I used to initialize the cluster using <code>kubeadm init --config=config.yml</code></p>
  
  <pre><code>apiServer:
  certSANs:
  - 10.62.194.4
  extraArgs:
      authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta2
  certificatesDir: /etc/kubernetes/pki
  clusterName: dev-cluster
  controllerManager: {}
  dns:
  type: CoreDNS
  etcd:
  local:
      dataDir: /var/lib/etcd
  imageRepository: k8s.gcr.io
  kind: ClusterConfiguration
  kubernetesVersion: v1.15.1
  networking:
  dnsDomain: cluster.local
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
  scheduler: {}
  </code></pre>
  
  <p>Result of <code>kubectl get svc --all-namespaces -o wide</code></p>
  
  <pre><code>NAMESPACE     NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
  default       service/kubernetes      ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP                  6d12h   &lt;none&gt;
  default       service/nginx-deploy    ClusterIP   10.97.5.194     &lt;none&gt;        80/TCP                   5d17h   run=nginx
  dmi           service/elasticsearch   ClusterIP   10.107.84.159   &lt;none&gt;        9200/TCP,9300/TCP        2d14h   app=dmi,component=elasticse
  dmi           service/metric-server   ClusterIP   10.106.117.2    &lt;none&gt;        8098/TCP                 2d14h   app=dmi,component=metric-se
  kube-system   service/calico-typha    ClusterIP   10.97.201.232   &lt;none&gt;        5473/TCP                 6d12h   k8s-app=calico-typha
  kube-system   service/kube-dns        ClusterIP   10.96.0.10      &lt;none&gt;        53/UDP,53/TCP,9153/TCP   6d12h   k8s-app=kube-dns
  kube-system   service/tiller-deploy   ClusterIP   10.98.133.94    &lt;none&gt;        44134/TCP                3d9h    app=helm,name=tiller
  </code></pre>
  
  <p>The command I ran was <code>kubectl exec -ti dnstools -- curl 10.96.0.10:53</code></p>
  
  <p>EDIT: </p>
  
  <p>I raised this question because I got this error when trying to resolve service names from within the cluster. I was under the impression that I got this error because I cannot ping the DNS server from a pod.</p>
  
  <p>Output of <code>kubectl exec -ti dnstools -- nslookup kubernetes.default</code></p>
  
  <pre><code>;; connection timed out; no servers could be reached
  
  command terminated with exit code 1
  </code></pre>
  
  <p>Output of <code>kubectl exec dnstools cat /etc/resolv.conf</code></p>
  
  <pre><code>nameserver 10.96.0.10
  search default.svc.cluster.local svc.cluster.local cluster.local reddog.microsoft.com
  options ndots:5
  </code></pre>
  
  <p>Result of <code>kubectl get ep kube-dns --namespace=kube-system</code></p>
  
  <pre><code>NAME       ENDPOINTS                                                  AGE
  kube-dns   192.168.0.2:53,192.168.0.3:53,192.168.0.2:53 + 3 more...   6d13h
  </code></pre>
  
  <p>EDIT:
  Ping-ing the CoreDNS pod directly using its Pod IP times out as well:</p>
  
  <pre><code>/ # ping 192.168.0.2
  PING 192.168.0.2 (192.168.0.2): 56 data bytes
  ^C
  --- 192.168.0.2 ping statistics ---
  24 packets transmitted, 0 packets received, 100% packet loss
  </code></pre>
  
  <p>EDIT:
  I think something has gone wrong when I was setting up the cluster. Below are the steps I took when setting up the cluster:</p>
  
  <ol>
  <li>Edit host files on master and worker to include the IP's and hostnames of the nodes</li>
  <li>Disabled swap using <code>swapoff -a</code> and disabled swap permanantly by editing <code>/etc/fstab</code></li>
  <li>Install docker prerequisites using <code>apt-get install apt-transport-https ca-certificates curl software-properties-common -y</code></li>
  <li>Added Docker GPG key using <code>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -</code></li>
  <li>Added Docker repo using <code>add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"</code></li>
  <li>Install Docker using <code>apt-get update -y; -get install docker-ce -y</code></li>
  <li>Install Kubernetes prerequisites using <code>curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -</code></li>
  <li>Added Kubernetes repo using <code>echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list</code></li>
  <li>Update repo and install Kubernetes components using <code>apt-get update -y; apt-get install kubelet kubeadm kubectl -y</code></li>
  </ol>
  
  <p>Configure master node:</p>
  
  <ol>
  <li><code>kubeadm init --apiserver-advertise-address=10.62.194.4 --apiserver-cert-extra-sans=10.62.194.4  --pod-network-cidr=192.168.0.0/16</code></li>
  <li>Copy Kube config to <code>$HOME</code>: <code>mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config</code></li>
  <li>Installed Calico using <code>kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml; kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml</code></li>
  </ol>
  
  <p>On node:
  On the node I did the <code>kubeadm join</code> command using the command printed out from <code>kubeadm token create --print-join-command</code> on the master</p>