<p>My application has a <code>/health</code> Http endpoint, configured for Kubernetes liveness check probe. The API returns a json, containing the health indicators.</p>
<p>Kubernetes only cares about the returned http status, but I would like to store the json responses in Prometheus for monitoring purposes.</p>
<p>Is it possible to catch the response once Kubernetes calls the API? I do not want to add the feature to the application itself but use an external component.</p>
<p>What is the recommended way of doing it?</p>

<p><strong>Answering to what you've asked:</strong></p>
<ol>
<li><p>Make a sidecar that calls <code>localhost:port/health</code> every <code>N</code> seconds and stores the most recent reply. <code>N</code> should be equal to the prometheus scraping interval for accurate results.</p>
</li>
<li><p>A sidecar then exposes the most recent reply in the form of metric in <code>/metrics</code> endpoint, on a separate port of a pod. You could use <a href="https://github.com/prometheus/client_python" rel="nofollow noreferrer">https://github.com/prometheus/client_python</a> to implement the sidecar. <em>Prometheus exporter sidecar</em> is actually a widely used pattern, try searching for it.</p>
</li>
<li><p>Point Prometheus to service <code>/metrics</code> endpoint, which is now served by a sidecar, on a separate port. You will need a separate port in a <code>Service</code> object, to point to your sidecar port. The scraping interval can be adjusted at this stage, to be in sync with your <code>N</code>, or otherwise, just adjust <code>N</code>. For <code>scrape_config</code> details refer to: <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#service" rel="nofollow noreferrer">https://prometheus.io/docs/prometheus/latest/configuration/configuration/#service</a></p>
<p>If you need an automatic Prometheus target discovery, say you have a bunch of deployments like this and their number varies - refer to my recent answer: <a href="https://stackoverflow.com/a/64269434/923620">https://stackoverflow.com/a/64269434/923620</a></p>
</li>
</ol>
<p><strong>Proposing a simpler solution:</strong></p>
<ol>
<li>In your app ensure you log everything you return in <code>/health</code></li>
<li>Implement centralized logging (log aggregation): <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/cluster-administration/logging/</a></li>
<li>Use a log processor, e.g. ELK, to query/analyze the results.</li>
</ol>
