<p>You can get events with <code>Readiness probe failed ...</code> but there are two problems with this:</p>
<ol>
<li>a readiness probe fail doesn't mean that the pod becomes not-ready since it may require more than one probe fail (<code>readinessProbe.failureThreshold</code>)</li>
<li>k8s will aggregate / summarize them these events making it almost impossible to understand and debug specific cases</li>
</ol>
<p>Is there a way to get this information with (<code>kubectl</code> and co) and without development code (like a custom resource)?</p>

rfalke
<p>Sadly, you won't be able to find Kubernetes <code>events</code> for pods becoming ready or not ready, because the status change is not an event, just an update made by Kubelet :/</p>
<p>The probes itself are indeed <a href="https://github.com/kubernetes/kubernetes/blob/v1.28.0/pkg/kubelet/prober/prober.go#L100-L117" rel="nofollow noreferrer">issuing events</a>, but as you've mentioned these are just &quot;Readiness probe failed&quot; logs, that you can't really use. As a side note, even if you were able to use these events, you will be limited by the fact Kubernetes <a href="https://github.com/kubernetes/kubernetes/blob/v1.28.0/pkg/kubelet/prober/prober.go#L116" rel="nofollow noreferrer">does not emit events when readiness probe succeeds</a>.</p>
<p>Anyway, there are 3 ways (probably more, but this should be enough here) to find out when <code>Ready</code> condition changes.</p>
<h3>Watch your pod(s) conditions with kubectl</h3>
<p>This requires to have a script running while your pod is running. With <code>kubectl</code>:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl get pods -n default name-of-my-pod --watch-only --no-headers \
   -o jsonpath='{&quot;Ready=&quot;}{.status.conditions[?(@.type==&quot;Ready&quot;)].status}{&quot;\n&quot;}' | \
   while read line ; \
   do \
     echo -e &quot;$(date --rfc-3339=ns) $line&quot; ; \
   done
</code></pre>
<p>The first part will watch the pod named <code>name-of-my-pod</code> and only outputs the <code>Ready</code> condition status. The second part of the script is to add a timestamp as prefix, as <code>kubectl ... --watch</code> does not provide an option to print timestamps for now (<a href="https://stackoverflow.com/a/63774000">more details here</a>).</p>
<p>The result will look like this:</p>
<pre><code>2023-08-21 16:28:20.866158908+02:00 Ready=True
2023-08-21 16:28:21.579293747+02:00 Ready=False
2023-08-21 16:28:24.592714416+02:00 Ready=True
2023-08-21 16:28:27.562872181+02:00 Ready=False
</code></pre>
<p>Obviously, you can follow the pod status in real-time, but if the script exits, you won't be able to find the history. But it should be enough for most real-time debugging use cases.</p>
<h3>Parse kubelet logs</h3>
<p>This requires to first add more verbosity to kubelet logs (with <code>--v=3</code> option). Once done, after retrieving kubelet logs, you'll be able to find the info about the status of the pod by looking at the <code>Status for pod updated successfully</code> logs:</p>
<pre class="lang-bash prettyprint-override"><code>cat kubelet.log | \
  grep 'pod=&quot;default/name-of-my-pod&quot;' | \
  grep 'Status for pod updated successfully' | \
  sed 's~\(.\{15\}\).*Ready:\(true\|false\).*~\1 Ready=\2~'
</code></pre>
<p>The parsing command works for Kubernetes 1.25.3, but it can evolve in the future, and parsing with <code>grep</code> and <code>sed</code> makes this very fragile (e.g. as of now, the first 15 characters is the date, but what if that changes someday?).</p>
<p>Anyway, the result will look like this:</p>
<pre><code>Aug 21 14:28:20 Ready=true
Aug 21 14:28:21 Ready=false
Aug 21 14:28:24 Ready=true
Aug 21 14:28:27 Ready=false
</code></pre>
<p>The main advantage over the <code>kubectl</code> solution is that you can find the history, and don't need to run a script in the background.</p>
<h3>Use a dedicated monitoring tool</h3>
<p>The most robust solution is to use a dedicated monitoring tool, like <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a>, to scrape your Kubernetes cluster metrics. You'll probably need this anyway in the future for other debugging needs, so better use it right now for your use-case. If you have it already installed, great! Otherwise, this <a href="https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-48.3.2/charts/kube-prometheus-stack" rel="nofollow noreferrer">Helm chart</a> can help you install it very quickly.</p>
<p>The <code>kube_pod_status_ready</code> metric in <a href="https://github.com/kubernetes/kube-state-metrics/blob/v2.9.2/docs/pod-metrics.md" rel="nofollow noreferrer"><code>kube-state-metrics</code> pod metrics</a> should be enough. As long as you have a short <code>scrape_interval</code> configured for Prometheus, you should be able to see every change of your pod <code>Ready</code> condition.</p>
<p>The following PromQL query:</p>
<pre><code>kube_pod_status_ready{namespace=&quot;default&quot;, pod=&quot;name-of-my-pod&quot;, condition=&quot;true&quot;}
</code></pre>
<p>will give you the following result:</p>
<p><a href="https://i.stack.imgur.com/VPFqQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VPFqQ.png" alt="PromQL query result to retrieve the pod Ready status" /></a></p>
<p>When <code>value = 1</code>, the pod is ready; otherwise, it's not ready.</p>
<p>Here, my scrape interval is 1s, so I have a less than 1s delay between the pod passes ready and the scrape. But if you have a scrape interval of 30s, be careful that the moment it shows ready/not-ready in the graph is probably not the exact moment it has passed ready/not-ready.</p>
<p>Also, with Prometheus, you'll be able to retrieve the status in the past (depending of your configuration and retention of metrics).</p>
