<p>I've deployed Prometheus, Alertmanager, Grafana et cetera in the <code>cattle-monitoring-system</code> namespace using the rancher-monitoring Helm chart. Look <a href="https://github.com/rancher/charts/tree/release-v2.5/charts/rancher-monitoring" rel="nofollow noreferrer">here</a> and <a href="https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/" rel="nofollow noreferrer">here</a> for documentation. Everything in this namespace is running OK.</p>
<p>Now I'm trying to deploy an extra Prometheus instance in another namespace called <code>indigo</code> using the manifest files. Unfortunately the configuration scrape_configs file won't load. I checked this by running the command <code>kubectl exec -it prometheus-indigo-0 -c prometheus -n indigo -- cat /etc/prometheus/config_out/prometheus.env.yaml</code>. Running the same command in the other Prometheus instance will return all the scrape_configs configuration.</p>
<p>My <code>deployment.yml</code> file looks like this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitoring-rancher-monitor-prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: indigo
  namespace: $NAMESPACE
spec:
  serviceAccountName: prometheus
  additionalScrapeConfigs:
    name: prometheus-scrape-configs-secret
    key: prometheus-scrape-configs.yml
  resources:
    requests:
      memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-indigo
  namespace: $NAMESPACE
spec:
  type: NodePort
  ports:
  - name: web
    nodePort: 30900
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: indigo
</code></pre>
<p>The <code>prometheus-scrape-configs.yml</code> file looks like this:</p>
<pre class="lang-yaml prettyprint-override"><code>global:
  evaluation_interval: 30s
  scrape_interval: 30s
scrape_configs:
- job_name: blackbox # To get metrics about the exporter itself
  metrics_path: /metrics
  static_configs:
    - targets:
      - .....

- job_name: blackbox-http # To get metrics about the exporter’s targets
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - .....
      labels:
        env: elise
    - targets:
      - .....
      labels:
        env: osb
    - targets:
      - .....
      labels:
        env: itp
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: .....
</code></pre>
<p>Finally, the <code>prometheus-scrape-configs-secret</code> is created in the <code>Makefile</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>include ../../../common.mk
include ../../environments/$(ENVIRONMENT).mk

NAMESPACE ?= indigo

.PHONY: deploy
deploy: \
    init \
    deploy-monitoring

.PHONY: init
init:

.PHONY: deploy-monitoring
deploy-monitoring:
    kubectl create secret generic prometheus-scrape-configs-secret \
    -n $(NAMESPACE) --from-file=prometheus-scrape-configs.yml

    envsubst &lt; $(ENVIRONMENT)-deployment.yml | kubectl apply -f -

.PHONY: delete
delete: delete-monitoring

.PHONY: delete-monitoring
delete-monitoring:
    kubectl delete secret prometheus-scrape-configs-secret -n $(NAMESPACE)
    envsubst &lt; $(ENVIRONMENT)-deployment.yml | kubectl delete -f -
</code></pre>
<p>So does anybody know why the <code>scrape_configs</code> file is unable to load correctly?</p>
<p>EDIT:</p>
<p>After running the command <code>kubectl describe pod prometheus-indigo-0 -n indigo</code> I noticed this error below. I also noticed after deploying this Prometheus instance it will get one error, immediately restarts and after that it is running..</p>
<pre class="lang-yaml prettyprint-override"><code>State:       Running
  Started:   Tue, 14 Jun 2022 18:01:23 +0200
Last State:  Terminated
  Reason:    Error
  Message:   ts=2022-06-14T16:01:19.743Z caller=main.go:450 level=error msg=&quot;Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)&quot; file=/etc/prometheus/config_out/prometheus.env.yaml err=&quot;open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory&quot;

</code></pre>

<p><a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#:%7E:text=The%20Prometheus%20resource%20declaratively%20describes,of%20ServiceMonitors%20to%20be%20used." rel="nofollow noreferrer">Thanks to this documentation</a> I managed to fix this issue. I forgot to create a ServiceMonitor that's referring to the <code>blackbox-exporter</code> through labels. Eventually you need to link this in the Kubernetes Prometheus object using the <code>serviceMonitorSelector</code>. <a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md" rel="nofollow noreferrer">You can look here for more information</a> how to configure <code>serviceMonitorSelector</code></p>
