<p>I'm trying to deploy Prometheus on GKE to monitor an existing Strimzi Kafka GKE cluster, and am facing issues.
(ref - <a href="https://strimzi.io/docs/operators/latest/deploying.html#proc-metrics-deploying-prometheus-operator-str" rel="nofollow noreferrer">https://strimzi.io/docs/operators/latest/deploying.html#proc-metrics-deploying-prometheus-operator-str</a>)</p>
<p>Here is what is done :</p>
<ul>
<li>created a namespace - monitoring, while Kafka is deployed in namespace - kafka</li>
<li>modified the kafka-deployment.yaml to include metricsConfig and KafkaExporter
as specified in file <a href="https://github.com/strimzi/strimzi-kafka-operator/tree/0.26.0/examples/metrics/kafka-metrics.yaml" rel="nofollow noreferrer">https://github.com/strimzi/strimzi-kafka-operator/tree/0.26.0/examples/metrics/kafka-metrics.yaml</a></li>
</ul>
<p>here are the changes :</p>
<pre><code>apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prom-graf #1
spec:
  kafka:
    version: 3.0.0
    ........
     metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml 
     zookeeper:
      ......
      metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml    
    entityOperator: #11
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: &quot;.*&quot;
    groupRegex: &quot;.*&quot; 
 
</code></pre>
<p>Command to apply changes to Kafka cluster :</p>
<pre><code>kubectl apply -f kafka-deployment.yaml -n kafka
</code></pre>
<ol start="2">
<li>created the Prometheous operator</li>
</ol>
<pre><code>command :
curl -s https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml &gt; bundle.yaml
</code></pre>
<p>changed the namespace to 'monitoring' in bundle.yaml and created the Prometheous operator</p>
<pre><code>kubectl create -f bundle.yaml -n monitoring 
</code></pre>
<p>Ran the following commands to create the Prometheus &amp; Grafana on GKE:
files are the same as in github - <a href="https://github.com/strimzi/strimzi-kafka-operator/tree/0.26.0/examples/metrics" rel="nofollow noreferrer">https://github.com/strimzi/strimzi-kafka-operator/tree/0.26.0/examples/metrics</a>, although i've mad the namespace changes</p>
<pre><code>kubectl apply -f strimzi-pod-monitor.yaml  
(changed namespace to kafka, since kafka is deployed in namespace kafka)
kubectl apply -f prometheus-rules.yaml -n monitoring
kubectl apply -f prometheus.yaml -n monitoring
kubectl apply -f grafana.yaml - monitoring

- created secret from file - prometheus-additional.yaml
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n monitoring 

</code></pre>
<p>When i inspect the object created on namespace - monitoring, here is what is listed :</p>
<pre><code>Karans-MacBook-Pro:prometheus-yamls karanalang$ kc get all,prometheus -n monitoring
NAME                                       READY   STATUS    RESTARTS   AGE
pod/grafana-5ccfb68647-55mzk               0/1     Running   0          21s
pod/prometheus-operator-85d84bb848-57lrn   1/1     Running   0          46m

NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/grafana               ClusterIP   10.100.3.10   &lt;none&gt;        3000/TCP   21s
service/prometheus-operator   ClusterIP   None          &lt;none&gt;        8080/TCP   46m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana               0/1     1            0           24s
deployment.apps/prometheus-operator   1/1     1            1           46m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5ccfb68647               1         1         0       23s
replicaset.apps/prometheus-operator-85d84bb848   1         1         1       46m

NAME                                          VERSION   REPLICAS   AGE
prometheus.monitoring.coreos.com/prometheus             3          44m
</code></pre>
<p>Question is - how do i get the hostname for the Prometheus install (since i need this to add as source in Grafana)?
Somehow, i don't see the prometheus pod, should that not be created ?</p>
<p>any ideas how to debug/fix this ?
tia !</p>

Karan Alang
<p>Assuming that the Prometheus pods did start, their collective hostnames would be found via service discovery like <code>prometheus.monitoring.svc.cluster.local</code></p>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/service-discovery" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/concepts/service-discovery</a></p>
<p>You might also be interesting in <a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#expose-the-prometheus-instance" rel="nofollow noreferrer">exposing Prometheus</a> itself</p>
