<p>I created a custom exporter in Python using prometheus-client package. Then created necessary artifacts to find the metric as a target in Prometheus deployed on Kubernetes.</p>
<p>But I am unable to see the metric as a target despite following all available instructions.
Help in finding the problem is appreciated.</p>
<p>Here is the summary of what I did.</p>
<ol>
<li>Installed Prometheus using Helm on the K8s cluster in a namespace <em><strong>prometheus</strong></em></li>
<li>Created a python program with prometheus-client package to create a metric</li>
<li>Created and deployed an image of the exporter in dockerhub</li>
<li>Created a deployment against the metrics image, in a namespace <em><strong>prom-test</strong></em></li>
<li>Created a Service, ServiceMonitor, and a ServiceMonitorSelector</li>
<li>Created a service account, role and binding to enable access to the end point</li>
</ol>
<p>Following is the code.</p>
<p><em><strong>Service &amp; Deployment</strong></em></p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: test-app-exporter
  namespace: prom-test
  labels:
    app: test-app-exporter
spec:
  type: ClusterIP
  selector:
    app: test-app-exporter
  ports:
  - name: http
    protocol: TCP
    port: 6000
    targetPort: 5000
    

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-exporter
  namespace: prom-test
spec:
  selector:
    matchLabels:
      app: test-app-exporter
  template:
    metadata:
      labels:
        app: test-app-exporter
    spec:
      #serviceAccount: test-app-exporter-sa
      containers:
      - name: test-app-exporter
        image: index.docker.io/cbotlagu/test-app-exporter:2
        imagePullPolicy: Always
        resources:
          limits:
            memory: &quot;128Mi&quot;
            cpu: &quot;500m&quot;
        ports:
        - name: http
          containerPort: 5000
      imagePullSecrets:
        - name: myregistrykey    
</code></pre>
<p><em><strong>Service account and role binding</strong></em></p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-app-exporter-sa
  namespace: prom-test

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-app-exporter-binding
subjects:
  - kind: ServiceAccount
    name: test-app-exporter-sa
    namespace: prom-test
roleRef:
  kind: ClusterRole
  name: test-app-exporter-role
  apiGroup: rbac.authorization.k8s.io

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-app-exporter-role
rules:
- apiGroups: [&quot;&quot;]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- apiGroups: [&quot;&quot;]
  resources:
  - configmaps
  verbs: [&quot;get&quot;]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
</code></pre>
<p><em><strong>Service Monitor &amp; Selector</strong></em></p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: test-app-exporter-sm
  namespace: prometheus
  labels:
    app: test-app-exporter
    release: prometheus
spec:
  selector:
    matchLabels:
      # Target app service
      app: test-app-exporter
  endpoints:
  - port: http
    interval: 15s
    path: /metrics
  namespaceSelector:
    matchNames:
    - default
    - prom-test
    #any: true
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: service-monitor-selector
  namespace: prometheus
spec:
  serviceAccountName: test-app-exporter-sa
  serviceMonitorSelector:
    matchLabels:
      app: test-app-exporter-sm
      release: prometheus
  resources:
    requests:
      memory: 400Mi
</code></pre>

Padmaja
<p>I am able to get the target identified by Prometheus.
But though the end point can be reached within the cluster as well as from the node IP. Prometheus says the target is down.
In addition to that I am unable to see any other target.
<a href="https://i.stack.imgur.com/h1VKm.png" rel="nofollow noreferrer">Prom-UI</a></p>
<p><strong>Any help is greatly appreciated</strong></p>
<p>Following is my changed code
<strong>Deployment &amp; Service</strong></p>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: prom-test
---  
apiVersion: v1
kind: Service
metadata:
  name: test-app-exporter
  namespace: prom-test
  labels:
    app: test-app-exporter
spec:
  type: NodePort
  selector:
    app: test-app-exporter
  ports:
  - name: http
    protocol: TCP
    port: 5000
    targetPort: 5000
    

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-exporter
  namespace: prom-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app-exporter
  template:
    metadata:
      labels:
        app: test-app-exporter
    spec:
      serviceAccountName: rel-1-kube-prometheus-stac-operator
      containers:
      - name: test-app-exporter
        image: index.docker.io/cbotlagu/test-app-exporter:2
        imagePullPolicy: Always
        resources:
          limits:
            memory: &quot;128Mi&quot;
            cpu: &quot;500m&quot;
        ports:
        - name: http
          containerPort: 5000
      imagePullSecrets:
        - name: myregistrykey    
</code></pre>
<p><strong>Cluster Roles</strong></p>
<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: prom-test
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - nodes/metrics
  - endpoints
  - pods
  - services
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-from-prom-test
  namespace: prom-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
subjects:
- kind: ServiceAccount
  name: rel-1-kube-prometheus-stac-operator
  namespace: monitoring
---  
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: monitoring-role
  namespace: monitoring
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - nodes/metrics
  - endpoints
  - pods
  - services
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-from-prom-test
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: monitoring-role
subjects:
- kind: ServiceAccount
  name: rel-1-kube-prometheus-stac-operator
  namespace: monitoring
</code></pre>
<p><strong>Service Monitor</strong></p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: test-app-exporter-sm
  namespace: monitoring
  labels:
    app: test-app-exporter
    release: prometheus
spec:
  selector:
    matchLabels:
      # Target app service
      app: test-app-exporter
  endpoints:
  - port: http
    interval: 15s
    path: /metrics
  namespaceSelector:
    matchNames:
    - prom-test
    - monitoring
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: service-monitor-selector
  namespace: monitoring
spec:
  serviceAccountName: rel-1-kube-prometheus-stac-operator
  serviceMonitorSelector:
    matchLabels:
      app: test-app-exporter
      release: prometheus
  resources:
    requests:
      memory: 400Mi
</code></pre>
