title: How to distinguish metrics from different services
body: <p>I'm playing with OpenTelemetry and have such a setup:</p>
  <p><em>Golang, docker-compose, 3 services, 1 standalone open-telemetry collector, 1 Prometheus.</em></p>
  <p><a href="https://i.stack.imgur.com/UVdb6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UVdb6.png" alt="enter image description here" /></a></p>
  <p>I collect some system metrics to a standalone open-telemetry collector. These metrics are collected from 3 different services and metrics have identical names. Then, Prometheus gets the data from the open-telemetry collector. The problem is that I can't distinguish metrics from different services in Prometheus because all of the metrics have the same &quot;instance&quot; value, which is equal to the open-telemetry-collector's host.</p>
  <p>I know that I can add a label with a service's name to the metric record and then distinguish the metrics by the label, but I'm searching for another solution because it is not always possible to add the label to each metric. Maybe, something like http-middleware, but for metrics, or maybe something on an infrastructure level.</p>
  <p>Services are written with Golang, but I will be glad to see the solution in any other language.</p>
  <p>otel-collector-config:</p>
  <pre><code>receivers:
    otlp:
      protocols:
        grpc:
        http:
  
  exporters:
    prometheus:
      endpoint: otel-collector:8889
      const_labels:
        label1: value1
      send_timestamps: true
      metric_expiration: 5m
  
  processors:
    batch:
  
  service:
    pipelines:
      metrics:
        receivers: [ otlp ]
        processors: [ batch ]
        exporters: [ prometheus ]
  </code></pre>
  <p>Prometheus config:</p>
  <pre><code>scrape_configs:
    - job_name: 'otel-collector'
      scrape_interval: 5s
      static_configs:
        - targets: ['otel-collector:8889']
  </code></pre>
  <p>docker-compose:</p>
  <pre><code>version: &quot;3.9&quot;
  services:
  
    service1:
      build:
        context: ./service1
        network: host
      environment:
        - TELEMETRY_COLLECTOR_ADDR=otel-collector:55681
      ports:
        - &quot;8094:8080&quot;
      expose:
        - &quot;8080&quot;
  
    service2:
      build:
        context: ./service2
        network: host
      environment:
        - TELEMETRY_COLLECTOR_ADDR=otel-collector:55681
      ports:
        - &quot;8095:8080&quot;
      expose:
        - &quot;8080&quot;
  
    service3:
      build:
        context: ./service3
        network: host
      environment:
        - TELEMETRY_COLLECTOR_ADDR=otel-collector:55681
      expose:
        - &quot;8080&quot;
      ports:
        - &quot;8096:8080&quot;
  
    prometheus:
      image: prom/prometheus:v2.26.0
      volumes:
        - ./prometheus.yaml:/etc/prometheus/prometheus.yml
      ports:
        - &quot;9090:9090&quot;
  
    otel-collector:
      image: otel/opentelemetry-collector:0.23.0
      command: [ &quot;--config=/etc/otel-collector-config.yaml&quot; ]
      expose:
        - &quot;55681&quot; # HTTP otel receiver
        - &quot;8889&quot; # Prometheus exporter metrics
      volumes:
        - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
  </code></pre>
  <p><strong>Update 1.</strong>
  I found that some new parameters were added to exporter-config <a href="https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper" rel="nofollow noreferrer">https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper</a> . One of them is what suitable for me: <code>resource_to_telemetry_conversion</code>. But as I see <code>prometheusexporter</code> and <code>prometheusremotewriteexporter</code> don't support that field in the config.</p>