<p>I'd like to monitor my Kubernetes Service objects to ensure that they have > 0 Pods behind them in "Running" state.</p>

<p>However, to do this I would have to first group the Pods by service and then group them further by status. </p>

<p>I would also like to do this programatically (e.g. for each service in namespace ...)</p>

<p>There's already some code that does this in the Sensu kubernetes plugin: <a href="https://github.com/sensu-plugins/sensu-plugins-kubernetes/blob/master/bin/check-kube-service-available.rb" rel="nofollow noreferrer">https://github.com/sensu-plugins/sensu-plugins-kubernetes/blob/master/bin/check-kube-service-available.rb</a> but I haven't seen anything that shows how to do it with Prometheus.</p>

<p>Has anyone setup kubernetes service level health checks with Prometheus? If so, how did you group by service and then group by Pod status? </p>

<p>The examples I have seen for Prometheus service checks relied on the blackbox exporter:</p>

<p>The blackbox exporter will try a given URL on the service. If that succeeds, at least one pod is up and running.</p>

<p>See here for an example: <a href="https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml" rel="nofollow noreferrer">https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml</a> in job kubernetes-service-endpoints</p>

<p>The URL to probe might be your liveness probe or something else. If your services don't talk HTTP, you can make the blackbox exporter test other protocols as well. </p>
