<p>I have a simple service and pod as described below but the readiness probe fails complaining for connection refused</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: keystone-api
spec:
  selector:
    app: keystone
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
      name: public
    - protocol: TCP
      port: 35357
      targetPort: 35357
      name: admin
...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keystone
  labels:
    app: keystone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keystone
  template:
    metadata:
      labels:
        app: keystone
    spec:
      containers:
        - name: keystone
          image: openio/openstack-keystone
          readinessProbe:
            tcpSocket:
              port: 5000
          env:
            - name: OS_IDENTITY_ADMIN_PASSWD
              value: password
            - name: IPADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 5000
              name: public
            - containerPort: 35357
              name: admin
</code></pre>
<p>Error:</p>
<pre><code> Normal   Pulled     37m                kubelet, kind-pl  Successfully pulled image &quot;openio/openstack-keystone&quot;
  Normal   Created    37m                kubelet, kind-pl  Created container keystone
  Normal   Started    37m                kubelet, kind-pl  Started container keystone
  Warning  Unhealthy  35m (x8 over 37m)  kubelet, kind-pl  Readiness probe failed: dial tcp 10.244.0.10:5000: connect: connection refused
</code></pre>
<p>This is how I launched the deployment and service
kubectl apply -f application.yaml --namespace=heat</p>
<p>What am i  missing here? Service spec</p>
<pre><code>spec:
  clusterIP: 10.96.162.65
  ports:
  - name: public
    port: 5000
    protocol: TCP
    targetPort: 5000
  - name: admin
    port: 35357
    protocol: TCP
    targetPort: 35357
  selector:
    app: keystone
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
</code></pre>
<p>From my VM:
telnet 10.96.162.65 5000
Trying 10.96.162.65...</p>
<p>Kubectl describe pod logs:</p>
<pre><code>Namespace:    heat
Priority:     0
Node:         kind-control-plane/172.17.0.2
Start Time:   Sun, 19 Apr 2020 16:04:36 +0530
Labels:       app=keystone
              pod-template-hash=8587f8dc76
Annotations:  &lt;none&gt;
Status:       Running
IP:           10.244.0.10
IPs:
  IP:           10.244.0.10
Controlled By:  ReplicaSet/keystone-8587f8dc76
Containers:
  keystone:
    Container ID:   containerd://9888e62ac7df3f076bd542591a6413a0ef5b70be2c792bbf06e423b5dae89ca0
    Image:          openio/openstack-keystone
    Image ID:       docker.io/openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
    Ports:          5000/TCP, 35357/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 19 Apr 2020 16:08:01 +0530
    Ready:          True
    Restart Count:  0
    Readiness:      tcp-socket :5000 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      OS_IDENTITY_ADMIN_PASSWD:  password
      IPADDR:                     (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wf2bp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wf2bp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wf2bp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;

## Kubectl log podname logs:

10.244.0.10 - - [19/Apr/2020 11:14:33] &quot;POST /v3/auth/tokens HTTP/1.1&quot; 201 2161
2020-04-19 11:14:33.699 49 INFO keystone.common.wsgi [req-fc64c89f-724c-4838-bc34-3907a8f79041 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services/heat
2020-04-19 11:14:33.705 49 WARNING keystone.common.wsgi [req-fc64c89f-724c-4838-bc34-3907a8f79041 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] Could not find service: heat.: ServiceNotFound: Could not find service: heat.
10.244.0.10 - - [19/Apr/2020 11:14:33] &quot;GET /v3/services/heat HTTP/1.1&quot; 404 90
2020-04-19 11:14:33.970 49 INFO keystone.common.wsgi [req-3589e675-8818-4b82-ad7d-c944d9e2a232 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services?name=heat
10.244.0.10 - - [19/Apr/2020 11:14:34] &quot;GET /v3/services?name=heat HTTP/1.1&quot; 200 341
2020-04-19 11:14:34.210 49 INFO keystone.common.wsgi [req-492a3e9f-8892-4204-8ca9-c1465e28e709 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] POST http://10.244.0.10:35357/v3/endpoints
10.244.0.10 - - [19/Apr/2020 11:14:34] &quot;POST /v3/endpoints HTTP/1.1&quot; 201 360
10.244.0.10 - - [19/Apr/2020 11:14:38] &quot;GET / HTTP/1.1&quot; 300 267
2020-04-19 11:14:38.089 49 INFO keystone.common.wsgi [req-4c8952b3-7d5b-4ee3-9cf9-f736e1628448 - - - - -] POST http://10.244.0.10:35357/v3/auth/tokens
10.244.0.10 - - [19/Apr/2020 11:14:38] &quot;POST /v3/auth/tokens HTTP/1.1&quot; 201 2367
2020-04-19 11:14:38.737 49 INFO keystone.common.wsgi [req-ebd817f5-d473-4909-b04d-ff0e1d5badab - - - - -] POST http://10.244.0.10:35357/v3/auth/tokens
10.244.0.10 - - [19/Apr/2020 11:14:39] &quot;POST /v3/auth/tokens HTTP/1.1&quot; 201 2367
2020-04-19 11:14:39.635 49 INFO keystone.common.wsgi [req-b68139dc-c62f-4fd7-9cfc-e472a88b9022 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services/heat
2020-04-19 11:14:39.640 49 WARNING keystone.common.wsgi [req-b68139dc-c62f-4fd7-9cfc-e472a88b9022 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] Could not find service: heat.: ServiceNotFound: Could not find service: heat.
10.244.0.10 - - [19/Apr/2020 11:14:39] &quot;GET /v3/services/heat HTTP/1.1&quot; 404 90
2020-04-19 11:14:39.814 49 INFO keystone.common.wsgi [req-6562f24f-f032-4150-86d9-951318918871 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services?name=heat
10.244.0.10 - - [19/Apr/2020 11:14:39] &quot;GET /v3/services?name=heat HTTP/1.1&quot; 200 341
2020-04-19 11:14:40.043 49 INFO keystone.common.wsgi [req-6542d767-29bf-4c1a-bbd9-a81a72e106dc 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] POST http://10.244.0.10:35357/v3/endpoints
10.244.0.10 - - [19/Apr/2020 11:14:40] &quot;POST /v3/endpoints HTTP/1.1&quot; 201 362
</code></pre>
Have manually created heat service
<pre><code>[root@keystone-8587f8dc76-rthmn /]# openstack service list                                                          
+----------------------------------+--------------+---------------+
| ID                               | Name         | Type          |
+----------------------------------+--------------+---------------+
| ec5ad9402b3b46599f3f8862e79429b3 | keystone     | identity      |
| 625d8b82a67d472981789f10ba37c381 | openio-swift | object-store  |
| 415b33b5d45c48f6916d38f7b146953a | heat         | orchestration |
+----------------------------------+--------------+---------------+
</code></pre>

Saurabh Arora
<p><strong>TL;DR:</strong></p>
<p>I've made some tests, your docker image and deployment seems really fine ,I was able to log into the pod, it was running and listening on the port.</p>
<ul>
<li>The reason why your readiness probe was returning <code>Warning Unhealthy...: connection refused</code> was because <strong>it was not given enough time for the pod to start.</strong></li>
</ul>
<p>I edited your deployment with the following lines:</p>
<pre><code>          readinessProbe:
            tcpSocket:
              port: 5000
            initialDelaySeconds: 300
            periodSeconds: 30
</code></pre>
<hr />
<p><strong>Explanation:</strong></p>
<ul>
<li>From <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes" rel="noreferrer">Configuring Probes</a> Documentation:</li>
</ul>
<blockquote>
<p><code>initialDelaySeconds</code>: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.</p>
<p><code>periodSeconds</code>: How often (in seconds) to perform the probe. Default to 10s. Minimum value is 1s.</p>
</blockquote>
<p><strong>NOTE:</strong> During my tests I noticed that the pod takes about 5 minutes to be running, way longer than the default 10s, that's why I set it as 300 seconds.</p>
<p>Meaning that after 5 minutes the pod was serving on port 5000.</p>
<p>Add the <code>initialDelaySeconds</code> line to your deployment and you should be fine.</p>
<hr />
<p><strong>Here is my Reproduction:</strong></p>
<ul>
<li>Edited Deployment:</li>
</ul>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: keystone-api
spec:
  selector:
    app: keystone
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
      name: public
    - protocol: TCP
      port: 35357
      targetPort: 35357
      name: admin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keystone
  labels:
    app: keystone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keystone
  template:
    metadata:
      labels:
        app: keystone
    spec:
      containers:
        - name: keystone
          image: openio/openstack-keystone
          readinessProbe:
            tcpSocket:
              port: 5000
            initialDelaySeconds: 300
            periodSeconds: 30
          env:
            - name: OS_IDENTITY_ADMIN_PASSWD
              value: password
            - name: IPADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 5000
              name: public
            - containerPort: 35357
              name: admin
</code></pre>
<ul>
<li>Create the resource and wait:</li>
</ul>
<pre><code>$ kubectl get pods  -w
NAME                        READY   STATUS    RESTARTS   AGE
keystone-7fd895cfb5-kqnnn   0/1     Running   0          3m28s
ubuntu                      1/1     Running   0          113m
keystone-7fd895cfb5-kqnnn   1/1     Running   0          5m4s
</code></pre>
<ul>
<li>After 5min4s the container was running <code>1/1</code> and I <code>describe</code> the pod:</li>
</ul>
<pre><code>$ kubectl describe pod keystone-586b8948d5-c4lpq
Name:         keystone-586b8948d5-c4lpq
Namespace:    default
Priority:     0
Node:         minikube/192.168.39.39
Start Time:   Mon, 20 Apr 2020 15:02:24 +0000
Labels:       app=keystone
              pod-template-hash=586b8948d5
Annotations:  &lt;none&gt;
Status:       Running
IP:           172.17.0.7
IPs:
  IP:           172.17.0.7
Controlled By:  ReplicaSet/keystone-586b8948d5
Containers:
  keystone:
    Container ID:   docker://8bc14d2b6868df6852967c4a68c997371006a5d83555c500d86060e48c549165
    Image:          openio/openstack-keystone
    Image ID:       docker-pullable://openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
    Ports:          5000/TCP, 35357/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 20 Apr 2020 15:02:26 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      tcp-socket :5000 delay=300s timeout=1s period=30s #success=1 #failure=3
    Environment:
      OS_IDENTITY_ADMIN_PASSWD:  password
      IPADDR:                     (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kcw8c (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-kcw8c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kcw8c
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  &lt;unknown&gt;  default-scheduler  Successfully assigned default/keystone-586b8948d5-c4lpq to minikube
  Normal  Pulling    7m12s      kubelet, minikube  Pulling image &quot;openio/openstack-keystone&quot;
  Normal  Pulled     7m11s      kubelet, minikube  Successfully pulled image &quot;openio/openstack-keystone&quot;
  Normal  Created    7m11s      kubelet, minikube  Created container keystone
  Normal  Started    7m11s      kubelet, minikube  Started container keystone
</code></pre>
<p>As you can see now there is no error.</p>
<p>Let me know in the comments if you have any doubt.</p>
