<p>We have set up full Prometheus stack - Prometheus/Grafana/Alertmanager/Node Explorer/Blackbox exporter using community helm charts in our Kubernetes cluster. Monitoring stack is deployed in its own namespace and our main software, comprised of microservices is deployed in the default namespace. Alerting is operating fine however blackbox exporter is not scraping correctly metrics (I guess) and FIRING regularly false positive alerts. We use the last for probing our microservices HTTP liveness/readiness endpoints.</p>
<p>My configuration (in values.yaml) related to the issue looks like:</p>
<pre><code>- alert: InstanceDown
           expr: up == 0
           for: 5m
           annotations:
             title: 'Instance {{ $labels.instance }} down'
             description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
- alert: ExporterIsDown
           expr: up{job=&quot;prometheus-blackbox-exporter&quot;} == 0
           for: 5m
           labels:
             severity: warning
           annotations:
             summary: &quot;Blackbox exporter is down&quot;
             description: &quot;Blackbox exporter is down or not being scraped correctly&quot;
...
...
...
extraScrapeConfigs:  |
   - job_name: 'prometheus-blackbox-exporter'
     metrics_path: /probe
     params:
       module: [http_2xx]
     static_configs:
       - targets:
         - http://service1.default.svc.cluster.local:8082/actuator/health/liveness
         - http://service2.default.svc.cluster.local:8081/actuator/health/liveness
         - http://service3.default.svc.cluster.local:8080/actuator/health/liveness
     relabel_configs:
       - source_labels: [__address__]
         target_label: __param_target
       - source_labels: [__param_target]
         target_label: instance
       - target_label: __address__
         replacement: prometheus-blackbox-exporter:9115
</code></pre>
<p>These 2 alerts are firing on every hour but at that time endpoints are 100% reachable.</p>
<p>We're using the default prometheus-blackbox-exporter/values.yaml file:</p>
<pre><code>config:
  modules:
    http_2xx:
      prober: http
      timeout: 5s
      http:
        valid_http_versions: [&quot;HTTP/1.1&quot;, &quot;HTTP/2.0&quot;]
        no_follow_redirects: false
        preferred_ip_protocol: &quot;ip4&quot;
</code></pre>
<p>Mails accordingly look this way:</p>
<pre><code>5] Firing
Labels
alertname = InstanceDown
instance = http://service1.default.svc.cluster.local:8082/actuator/health/liveness
job = prometheus-blackbox-exporter
severity = critical
</code></pre>
<p>another type of email</p>
<pre><code>Labels
alertname = ExporterIsDown
instance = http://service1.default.svc.cluster.local:8082/actuator/health/liveness
job = prometheus-blackbox-exporter
severity = warning
Annotations
description = Blackbox exporter is down or not being scraped correctly
summary = Blackbox exporter is down
</code></pre>
<p>Another odd thing I noticed is that in Prometheus UI I don't see any probe_* metrics as shown here <a href="https://lapee79.github.io/en/article/monitoring-http-using-blackbox-exporter/" rel="nofollow noreferrer">https://lapee79.github.io/en/article/monitoring-http-using-blackbox-exporter/</a>
Not sure what we are doing wrong or missing to do but it's very annoying to get hundreds of false positive emails.</p>

<p>Answering my own question. It seems that I have typed:</p>
<pre><code>replacement: prometheus-blackbox-exporter:9115
</code></pre>
<p>but instead it must be the service name:</p>
<pre><code>replacement: stage-prometheus-blackbox-exporter:9115
</code></pre>
<p>According to documentation:</p>
<p><em>replacement: localhost:9115 # The blackbox exporter’s real hostname:port. For Windows and macOS replace with - host.docker.internal:9115</em></p>
<p>For Kubernetes though it should be blackbox-exporter's service name which is not well documented. Or at least I haven't found this anywhere.</p>
<p>To get the service:</p>
<pre><code>kubectl get svc -l app.kubernetes.io/name=prometheus-blackbox-exporter
</code></pre>
