<p>I am trying to monitor external service (which is exporter of cassandra metrics) in prometheus-operator. I installed prometheus-operator using helm 2.11.0. I installed it using this yaml:</p>

<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
</code></pre>

<p>and these commands on my kubernetes cluster:</p>

<pre><code>kubectl create -f rbac-config.yml
helm init --service-account tiller --history-max 200
helm install stable/prometheus-operator --name prometheus-operator --namespace monitoring
</code></pre>

<p>Next, basing on article:
<a href="https://devops.college/prometheus-operator-how-to-monitor-an-external-service-3cb6ac8d5acb" rel="noreferrer">how monitor to an external service</a></p>

<p>I tried to do steps described in it. As suggested I created Endpoints, Service and ServiceMonitor with label for existing Prometheus. Here are my yaml files:</p>

<pre><code>apiVersion: v1
kind: Endpoints
metadata:
  name: cassandra-metrics80
  labels:
    app: cassandra-metrics80
subsets:
- addresses:
  - ip: 10.150.1.80
  ports:
  - name: web
    port: 7070
    protocol: TCP
</code></pre>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: cassandra-metrics80
  namespace: monitoring
  labels:
    app: cassandra-metrics80
    release: prometheus-operator
spec:
  externalName: 10.150.1.80
  ports:
  - name: web
    port: 7070
    protocol: TCP
    targetPort: 7070
  type: ExternalName
</code></pre>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cassandra-metrics80
  labels:
    app: cassandra-metrics80
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: cassandra-metrics80
      release: prometheus-operator
    namespaceSelector:
      matchNames:
      - monitoring
  endpoints:
  - port: web
    interval: 10s
    honorLabels: true
</code></pre>

<p>And in prometheus service discovery page I can see:
<a href="https://i.stack.imgur.com/8M40z.png" rel="noreferrer"><img src="https://i.stack.imgur.com/8M40z.png" alt="Service Discovery"></a></p>

<p>That this service is not active and all labels are dropped.
I did a numerous things trying to fix this, like setting targetLabels. Trying to relabel the once that are discovered, as
described here: <a href="https://github.com/coreos/prometheus-operator/issues/1166#issuecomment-447404996" rel="noreferrer">prometheus relabeling</a> 
But unfortunately nothing works. What could be the issue or how can I investigate it better?</p>

<p>Ok, I found out that service should be in the same namespace as service monitor and endpoint, after that prometheus started to see some metrics from cassandra.</p>
