<p>Any idea how to go about this, can't find much clear info on google, to measure the Errors (40x, and 50x) on my service endpoints. My services are up and when I delete pods just for a test, I can see in the blackbox metrics that the prometheus gets and error, but not specified like 40x type or 50x.</p>
<p>Edit 1:</p>
<ul>
<li>Yes, I have setup my cluster, at this stage is experimental, I have set it up on a VirtualBox+Vagrant+K3s. I have created two simple services one front end one backend, and configured prometheus Jobs to discover the services and probe their uptime via Blackbox monitor. My goal is to get somehow some metrics on a grafana dashboard the measure the number of 40x or 50x errors for all the requests to these services within a period of time. Currently whats on my mind is measuring the number of 2xx and reporting only Non-2xx status codes but that would include more errors/status than 40x and 50x.</li>
</ul>
<p>Prometheus is deployed as a helm stack, same with the Blackbox monitor. Everything is deployed on the default namespace, because at this stage is just for testing on how to achieve this goal.</p>

<p>Based on <a href="https://groups.google.com/g/prometheus-users/c/QUY9NsLPsZk" rel="nofollow noreferrer">this topic</a>:</p>
<blockquote>
<p>Services in Kubernetes are kind of like load-balancers - they just route requests to underlying pods. The pods themselves actually contain the application that does the work and returns the status code.
You don't monitor kubernetes services  <em>per-se</em> for 4xx or 5xx errors, you need to monitor the underlying application itself.</p>
</blockquote>
<p>So, you need to create an architecture to monitoring your application. Prometheus only collects metrics and makes graphs out of it, it does not process anything by itself. Metrics must be exposed by the application. <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus/" rel="nofollow noreferrer">Here</a> you can find topic - Kubernetes monitoring with Prometheus, the ultimate guide. Is very comprehensive and explains perfectly how to monitor an application. For you, the most interesting part should be <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus/#services" rel="nofollow noreferrer">How to monitor a Kubernetes service with Prometheus</a>. You can also find there a <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/" rel="nofollow noreferrer">Prometheus Operator Tutorial</a>. It could help you with automation deployment for Prometheus, Alertmanager and Grafana.</p>
<p>Once you've installed everything, you'll be able to collect metrics. It is good practice to use <a href="https://prometheus.io/docs/practices/instrumentation/#use-labels" rel="nofollow noreferrer">lables</a>. This allows you to easily distinguish between different response codes from your application.</p>
<blockquote>
<p>For example, rather than <code>http_responses_500_total</code> and <code>http_responses_403_total</code>, create a single metric called <code>http_responses_total</code> with a <code>code</code> label for the HTTP response code. You can then process the entire metric as one in rules and graphs.</p>
</blockquote>
