<p>I was trying readiness probe in kubernetes with a springboot app. After the app starts, lets say after 60 seconds I fire <code>ReadinessState.REFUSING_TRAFFIC</code> app event.</p>
<p>I use port-forward for kubernetes service(Cluster-Ip) and checked /actuator/health/readiness and see
<code>&quot;status&quot;:&quot;OUT_OF_SERVICE&quot;</code> after 60 seconds.</p>
<p>I, then fire some GET/POST requests to service.</p>
<p>Expected:
Service unavailable message</p>
<p>Actual:
GET/POST endpoints return data as usual</p>
<p>Is the behavior expected. Please comment.</p>
<p>Sample liveness/readiness probe yaml</p>
<pre><code>        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Authorization
              value: Basic xxxxxxxxxxxxxx
            path: /actuator/health/liveness
            port: http
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 10
        name: sample-app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Authorization
              value: Basic xxxxxxxxxxxxxx
            path: /actuator/health/readiness
            port: http
            scheme: HTTP
          initialDelaySeconds: 140
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 10
</code></pre>

<p>This is expected behavior as:</p>
<ul>
<li><code>$ kubectl port-forward service/SERVICE_NAME LOCAL_PORT:TARGET_PORT</code></li>
</ul>
<p>is not considering the state of the <code>Pod</code> when doing a port forwarding (directly connects to a <code>Pod</code>).</p>
<hr />
<h3>Explanation</h3>
<p>There is already a great answer which pointed me on further investigation here:</p>
<ul>
<li><em><a href="https://stackoverflow.com/a/59941521/12257134">Stackoverflow.com: Answer: Does kubectl port-forward ignore loadBalance services?</a></em></li>
</ul>
<p>Let's assume that you have a <code>Deployment</code> with a <code>readinessProbe</code> (in this example probe will never succeed):</p>
<ul>
<li><code>$ kubectl get pods, svc</code> (redacted not needed part)</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>NAME                                     READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-64ff4d8749-7khht    0/1     Running   0          97m
pod/nginx-deployment-64ff4d8749-bklnf    0/1     Running   0          97m
pod/nginx-deployment-64ff4d8749-gsmml    0/1     Running   0          97m

NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/nginx                ClusterIP   10.32.31.105   &lt;none&gt;        80/TCP      97m
</code></pre>
<ul>
<li><code>$ kubectl describe endpoints nginx</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>Name:         nginx
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
Subsets:
  Addresses:          &lt;none&gt;
  NotReadyAddresses:  10.36.0.62,10.36.0.63,10.36.0.64 # &lt;-- IMPORTANT
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    &lt;unset&gt;  80    TCP

Events:  &lt;none&gt;
</code></pre>
<p>As you can see all of the <code>Pods</code> are not in <code>Ready</code> state and the <code>Service</code> will not send the traffic to it. This can be seen in a following scenario ( create a test <code>Pod</code> that will try to <code>curl</code> the <code>Service</code>):</p>
<ul>
<li><code>$ kubectl run -it --rm nginx-check --image=nginx -- /bin/bash</code></li>
<li><code>$ curl nginx.default.svc.cluster.local</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>curl: (7) Failed to connect to nginx.default.svc.cluster.local port 80: Connection refused
</code></pre>
<p>Using <code>kubectl port-forward</code>:</p>
<ul>
<li><code>$ kubectl port-forward service/nginx 8080:80</code></li>
<li><code>$ curl localhost:8080</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>&lt;-- REDACTED --&gt; 
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
&lt;p&gt;If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.&lt;/p&gt;
&lt;-- REDACTED --&gt;
</code></pre>
<p>More light can be shed on why it happened by getting more verbose output from the command:</p>
<ul>
<li><code>kubectl port-forward service/nginx 8080:80 -v=6</code> (the number can be higher)</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>I0606 21:29:24.986382    7556 loader.go:375] Config loaded from file:  /SOME_PATH/.kube/config
I0606 21:29:25.041784    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/services/nginx 200 OK in 51 milliseconds
I0606 21:29:25.061334    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/pods?labelSelector=app%3Dnginx 200 OK in 18 milliseconds
I0606 21:29:25.098363    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/pods/nginx-deployment-64ff4d8749-7khht 200 OK in 18 milliseconds
I0606 21:29:25.164402    7556 round_trippers.go:444] POST https://API_IP/api/v1/namespaces/default/pods/nginx-deployment-64ff4d8749-7khht/portforward 101 Switching Protocols in 62 milliseconds
Forwarding from 127.0.0.1:8080 -&gt; 80
Forwarding from [::1]:8080 -&gt; 80
</code></pre>
<p>What happened:</p>
<ul>
<li><code>kubectl</code> requested the information about the <code>Service</code>: <code>nginx</code></li>
<li><code>kubectl</code> used the <code>selector</code> associated with the <code>Service</code> and looked for <code>Pods</code> with the same <code>selector</code> (<code>nginx</code>)</li>
<li><code>kubectl</code> chose a single <code>Pod</code> and port-forwarded to it.</li>
</ul>
<p>The <code>Nginx</code> welcome page showed as the <code>port-forward</code> connected directly to a <code>Pod</code> and not to a <code>Service</code>.</p>
<hr />
<p>Additional reference:</p>
<ul>
<li><p><em><a href="https://github.com/kubernetes/kubernetes/issues/15180" rel="nofollow noreferrer">Github.com: Kubernetes: Issues: kubectl port-forward should allow forwarding to a Service</a></em></p>
</li>
<li><p><code>$ kubectl port-forward --help</code></p>
</li>
</ul>
<blockquote>
<p># Listen on port 8443 locally, forwarding to the targetPort of the service's port named &quot;https&quot; in a <strong>pod selected by
the service</strong></p>
<blockquote>
<p><code>kubectl port-forward service/myservice 8443:https</code></p>
</blockquote>
</blockquote>
