<p>I've installed <strong>Prometheus</strong> and <strong>Grafana</strong> to monitor my <strong>K8S cluster</strong> and <strong>microservices</strong> using <strong>helm charts</strong>:</p>
<pre><code>helm install monitoring prometheus-community/kube-promehteus-stack --values prometheus-values.yaml --version 16.10.0 --namespace monitoring --create-namespace
</code></pre>
<p>the content of <code>promehteus-values.yaml</code> is:</p>
<pre><code>prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchLabels:
        prometheus: devops
commonLabels:
  prometheus: devops
grafana:
  adminPassword: test123

</code></pre>
<p>Then I installed <code>kong-ingress-controller</code> using <code>helm-charts</code>:</p>
<pre><code>helm install kong kong/kong --namespace kong --create-namespace --values kong.yaml --set ingressController.installCRDs=false
</code></pre>
<p>the content of <code>kong.yaml</code> file is:</p>
<pre><code>podAnnotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/port: &quot;8100&quot;

</code></pre>
<p>I've also changed the value of <code>metricsBindAdress</code> in <code>kube-proxy</code> <strong>configmap</strong> to <code>0.0.0.0:10249</code> .</p>
<p>then I installed <code>kong prometheus plugin</code> using this <code>yaml file</code> :</p>
<pre><code>apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: prometheus
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: &quot;true&quot;
plugin: prometheus

</code></pre>
<p>My <code>kong endpoint</code> is :</p>
<p><code>$ kubectl edit endpoints -n kong kong-kong-proxy</code></p>
<pre><code># Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: &quot;2021-10-27T03:28:25Z&quot;
  creationTimestamp: &quot;2021-10-26T04:44:57Z&quot;
  labels:
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: &quot;2.6&quot;
    enable-metrics: &quot;true&quot;
    helm.sh/chart: kong-2.5.0
  name: kong-kong-proxy
  namespace: kong
  resourceVersion: &quot;6553623&quot;
  uid: 91f2054f-7fb9-4d63-8b65-be098b8f6547
subsets:
- addresses:
  - ip: 10.233.96.41
    nodeName: node2
    targetRef:
      kind: Pod
      name: kong-kong-69fd7d7698-jjkj5
      namespace: kong
      resourceVersion: &quot;6510442&quot;
      uid: 26c6bdca-e9f1-4b32-91ff-0fadb6fce529
  ports:
  - name: kong-proxy
    port: 8000
    protocol: TCP
  - name: kong-proxy-tls
    port: 8443
    protocol: TCP

</code></pre>
<p>Finally I wrote the <code>serviceMonitor</code> for <code>kong</code> like this :</p>
<pre><code>
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    prometheus: devops
  name: kong-sm
  namespace: kong
spec:
  endpoints:
  - interval: 30s
    port: kong-proxy
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - kong
  selector:
    matchLabels:
      app.kubernetes.io/instance: kong

</code></pre>
<p>After all of this ; the <code>targets</code> in <code>prometheus dashboard</code> looks like this:</p>
<p><a href="https://i.stack.imgur.com/GikR0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GikR0.png" alt="enter image description here" /></a></p>
<p>What did I miss/do wrong?</p>

samm13
<p>Let's take a look to the <strong>Kong deployment</strong> first <strong>(</strong> pay extra attention to the bottom of this file <strong>)</strong>:</p>
<p><code>kubectl edit deploy -n kong kong-kong</code> :</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    [...]
  creationTimestamp: &quot;2021-10-26T04:44:58Z&quot;
  generation: 1
  labels:
    app.kubernetes.io/component: app
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: &quot;2.6&quot;
    helm.sh/chart: kong-2.5.0
  name: kong-kong
  namespace: kong
  [...]
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: app
      app.kubernetes.io/instance: kong
      app.kubernetes.io/name: kong
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    [...]
   - env:
        [...]
        image: kong:2.6
        imagePullPolicy: IfNotPresent
        [...]
        name: proxy
        ports:
        - containerPort: 8000
          name: proxy
          protocol: TCP
        - containerPort: 8443
          name: proxy-tls
          protocol: TCP
         ############################################
         ## THIS PART IS IMPORTANT TO US :          #
         ############################################
        - containerPort: 8100 
          name: status
          protocol: TCP
        [...]

</code></pre>
<p>As you can see, in the <code>sepc.template.spec.env.ports</code> part we have <strong>3 ports</strong>, the <strong>8100</strong> will be used for get <strong>metrics</strong> so if you can't see this port in the <strong>kong endpoint</strong>, add it manually to the bottom of kong endpoint:</p>
<p><code>$ kubectl edit endpoints -n kong kong-kong-proxy</code> :</p>
<pre><code>apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: &quot;2021-10-26T04:44:58Z&quot;
  creationTimestamp: &quot;2021-10-26T04:44:57Z&quot;
  labels:
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: &quot;2.6&quot;
    enable-metrics: &quot;true&quot;
    helm.sh/chart: kong-2.5.0
  name: kong-kong-proxy
  namespace: kong
  resourceVersion: &quot;7160332&quot;
  uid: 91f2054f-7fb9-4d63-8b65-be098b8f6547
subsets:
- addresses:
  - ip: 10.233.96.41
    nodeName: node2
    targetRef:
      kind: Pod
      name: kong-kong-69fd7d7698-jjkj5
      namespace: kong
      resourceVersion: &quot;6816178&quot;
      uid: 26c6bdca-e9f1-4b32-91ff-0fadb6fce529
  ports:
  - name: kong-proxy
    port: 8000
    protocol: TCP
  - name: kong-proxy-tls
    port: 8443
    protocol: TCP
#######################################
##        ADD THE 8100 PORT HERE      #
#######################################
  - name: kong-status
    port: 8100
    protocol: TCP

</code></pre>
<p>Then save this file and change the <strong>serviceMonitor</strong> of <strong>kong</strong> like this <strong>(</strong> the <strong>port</strong> name is the <strong>same</strong> to the <strong>endpoint</strong> we added recently <strong>)</strong>:</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    prometheus: devops
  name: kong-sm
  namespace: kong
spec:
  endpoints:
  - interval: 30s
#############################################################################
##   THE NAME OF THE PORT IS SAME TO THE NAME WE ADDED TO THE ENDPOINT FILE #
#############################################################################
    port: kong-status
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - kong
  selector:
    matchLabels:
      app.kubernetes.io/instance: kong
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: kong
      app.kubernetes.io/version: &quot;2.6&quot;
      enable-metrics: &quot;true&quot;
      helm.sh/chart: kong-2.5.0
</code></pre>
<p>Apply the <strong>serviceMonitor</strong> yaml file and after few seconds <strong>Prometheus</strong> will detect it as a <strong>target</strong> and scrape kong's metrics successfully.</p>
