<p>after reading <a href="https://alexandrev.medium.com/how-to-change-the-names-of-your-metrics-in-prometheus-b78497efb5de" rel="nofollow noreferrer">this</a> article I'm trying to cleanup metrics going out of Spark 3.0.1. Here is my servicemonitor.yml file:</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: spark3
    release:  prometheus
  name: spark3-servicemonitor
spec:
  endpoints:
  - interval: 5s
    port: spark-ui
    path: /metrics/prometheus
    relabelings:
    # Rename metrics
      - sourceLabels: [__name__]
        targetLabel: __name__
        regex: 'metrics_spark_driver_.+_StreamingMetrics_([a-zA-Z_]{1,})_Value'
        replacement: 'spark_driver_$1'
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      spark-version: &quot;3&quot;
</code></pre>
<p>I expect the following transformation:
metrics_spark_driver_whateverappid_StreamingMetrics_streaming_lastCompletedBatch_totalDelay_Value -&gt; spark_driver_streaming_lastCompletedBatch_totalDelay however the relabelling does not seem to work. Could you please assist me on this subject?</p>

eugen-fried
<p>The <code>relabelings</code> must be named <code>metricRelabelings</code> according to the <a href="https://github.com/prometheus-operator/prometheus-operator/blob/552d86a2713f6653146674f3598955a375b93099/bundle.yaml#L10224" rel="noreferrer">spec</a>. Note that the yaml format of the ServiceMonitors does not use the same key names as the corresponding prometheus config (but it is still valid yaml).</p>
