<p>I'm working on a production <strong>K8s</strong> cluster with an HTTP-based application and I'd like to setup monitoring and alerting for HTTP errors. It's clear how to check the uptime of the service (using monitoring e.g. stackdriver), but absolutely not regarding HTTP failure rate.  </p>

<p>I've got an <code>nginx-ingress-controller</code> as an end-point (with external load balancer).</p>

<p>How to collect and view the metrics such as latency, HTTP failures, etc. from this load balancer?      </p>

<p>In particular I need to now, when <strong>HTTP response failure rate</strong> exceeds some percentage. </p>

<p>If you are looking at monitoring HTTP 4XX and 5XX errors for example I believe the best way is to aggregate the load balancer and the nginx ingress controller logs in some logging tool. If you are looking at open source solutions you could use something like <a href="https://github.com/elastic/elasticsearch" rel="nofollow noreferrer">Elasticsearch</a> with <a href="https://github.com/elastic/kibana" rel="nofollow noreferrer">Kibana</a> to visualize the errors over a time frame. To send the logs you can use a forwarder like <a href="https://fluentbit.io/" rel="nofollow noreferrer">fluent-bit</a> or <a href="https://www.fluentd.org/" rel="nofollow noreferrer">Fluentd</a>.</p>

<p>If you have a budget for a paid tool you can use a commercially available solution like:</p>

<ul>
<li><a href="https://www.loggly.com/" rel="nofollow noreferrer">Loggly</a></li>
<li><a href="https://docs.datadoghq.com/logs/" rel="nofollow noreferrer">Datadog logging</a></li>
<li><a href="https://www.papertrail.com/" rel="nofollow noreferrer">Papertrail</a></li>
<li>etc.</li>
</ul>

<p>Then you can set up alerts with any of these tools. For Elasticsearch you can use something like <a href="https://github.com/Yelp/elastalert" rel="nofollow noreferrer">elastalert</a></p>

<p>If you are using GCP you can also use their <a href="https://cloud.google.com/logging" rel="nofollow noreferrer">Logging tool</a>, create a custom metric, and alert on that metric.</p>

<p><a href="https://i.stack.imgur.com/kGq17.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/kGq17.png" alt="Logging"></a></p>

<p>Another alternative, but may not have the metrics that you are looking for is to use <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a> with an <a href="https://github.com/nginxinc/nginx-prometheus-exporter" rel="nofollow noreferrer">Nginx ingress Prometheus exporter</a> to monitor nginx metrics (it depends on what metrics you'd like to monitor) </p>
