<p>I have an IaaC project that use Terraform and Helm charts to deploy Prometheus (along with Grafana, Kubernetes and the platform app). I do have Kubernetes metrics on Prometheus. However, I realize that the configuration on deployed Prometheus doesn't really come from the Prometheus config file in the project. I am not sure if it's the problem with configMap or other configuration. Here are the files concerned in the project:</p>
<p><code>/terraform/kubernetes/files/prometheus_config_map.yaml</code></p>
<pre><code>global:
  scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
  static_configs:
  - targets: ['localhost:9090']
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_label_component]
    action: replace
    target_label: job
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
</code></pre>
<p><code>/terraform/kubernetes/prometheus.tf</code></p>
<pre><code>resource &quot;kubernetes_namespace&quot; &quot;prometheus&quot; {
  metadata {
    name = &quot;prometheus&quot;
  }
}

resource &quot;kubernetes_config_map&quot; &quot;prometheus_config&quot; {
  metadata {
    name      = &quot;prometheus-config&quot;
    namespace = &quot;prometheus&quot;
  }

  data = {
    &quot;prometheus.yml&quot; = file(&quot;${path.module}/files/prometheus_config_map.yaml&quot;)
  }
  depends_on = [
    kubernetes_namespace.prometheus
  ]
}

# Values documentation: https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml
resource &quot;helm_release&quot; &quot;prometheus&quot; {
  name        = &quot;prometheus&quot;
  repository  = local.helm_repositories.bitnami
  chart       = &quot;kube-prometheus&quot;
  version     = &quot;3.4.0&quot;
  namespace   = &quot;prometheus&quot;
  atomic      = true
  max_history = 5

  values = [
    file(&quot;${path.module}/helm_values/security.yaml.tpl&quot;),
    file(&quot;${path.module}/helm_values/prometheus.yaml&quot;)
  ]

  depends_on = [
    kubernetes_config_map.prometheus_config
  ]
}
</code></pre>
<p><code>/terraform/kubernetes/helm_values/prometheus.yaml</code></p>
<pre><code>prometheus:
  podMetadata:
    annotations:
      container.apparmor.security.beta.kubernetes.io/prometheus-operator: runtime/default
      seccomp.security.alpha.kubernetes.io/pod: runtime/default

nodeAffinityPreset:
  ## Node affinity type
  ## Allowed values: soft, hard
  ##
  type: &quot;hard&quot;
  ## Node label key to match
  ## E.g.
  ## key: &quot;kubernetes.io/e2e-az-name&quot;
  ##
  key: &quot;cloud.google.com/gke-nodepool&quot;
  ## Node label values to match
  ## E.g.
  ## values:
  ##   - e2e-az1
  ##   - e2e-az2
  ##
  values: [
    &quot;project-primary-pool&quot;
  ]

prometheus:  
  configMaps:
    - prometheus-config
</code></pre>
<p>In this file <code>/terraform/kubernetes/helm_values/prometheus.yaml</code>, I tried to remove the last <code>prometheus:</code> and moved <code>configMaps:</code> to the root as per this <a href="https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml" rel="nofollow noreferrer">documentation</a> but it broke the Prometheus configuration.
As you can see in the file <code>/terraform/kubernetes/files/prometheus_config_map.yaml</code>, the <code>scrape_interval</code> is <code>15s</code> but when I check on Prometheus UI config, all of the scrape_interval is 30s, therefore, for sure the configuration of deployed Prometheus doesn't come from this file. Therefore, I have no way to change the configuration such as scrape_interval</p>
<p><a href="https://i.stack.imgur.com/Fsm9v.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Fsm9v.png" alt="promConfig1" /></a></p>
<p>I also notice in this <a href="https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml" rel="nofollow noreferrer">documentation</a> that I am missing many configuration in the file <code>/terraform/kubernetes/helm_values/prometheus.yaml</code> but I am not certain of what to add. Could you please let me know how I can resolve this? Thank you in advance.</p>
<p><a href="https://i.stack.imgur.com/y9sNE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/y9sNE.png" alt="promCONFIGmount" /></a></p>

shuti
<p>in values.yaml if you check the line : 577</p>
<p><a href="https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml" rel="nofollow noreferrer">https://github.com/bitnami/charts/blob/master/bitnami/kube-prometheus/values.yaml</a></p>
<p>there is an option to mount the config map to the deployment.</p>
<blockquote>
<h2>ConfigMaps that should be mounted into the Prometheus Pods</h2>
<p>configMaps: []</p>
</blockquote>
<p>you can set the config map to the deployment and change the time of interval.</p>
<p>if it's not using the config map values it must be using the default config from volume or directly you can go inside the POD and check that once also to verify.</p>
