<p>I successfully deployed helm chart <a href="https://github.com/coreos/prometheus-operator/tree/master/helm/prometheus-operator" rel="nofollow noreferrer">prometheus operator</a>,  <a href="https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus" rel="nofollow noreferrer">kube-prometheus</a> and <a href="https://github.com/kubernetes/charts/tree/master/incubator/kafka]" rel="nofollow noreferrer">kafka</a> (tried both image danielqsj/kafka_exporter <code>v1.0.1</code> and  <code>v1.2.0</code>). </p>

<p>Install with default value mostly, rbac are enabled. </p>

<p>I can see 3 <code>up</code> nodes in Kafka target list in prometheus, but when go in Grafana, I can's see any kafka metric with <a href="https://grafana.com/dashboards/721" rel="nofollow noreferrer">kafka overview</a></p>

<p>Anything I missed or what I can check to fix this issue?</p>

<p>I can see metrics start with <code>java_</code>, <code>kafka_</code>, but no <code>jvm_</code> and only few <code>jmx_</code> metrics.</p>

<p><a href="https://i.stack.imgur.com/mq9tA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mq9tA.png" alt="enter image description here"></a></p>

<p>I found someone reported similar issue (<a href="https://groups.google.com/forum/#!searchin/prometheus-users/jvm_%7Csort:date/prometheus-users/OtYM7qGMbvA/dZ4vIfWLAgAJ" rel="nofollow noreferrer">https://groups.google.com/forum/#!searchin/prometheus-users/jvm_%7Csort:date/prometheus-users/OtYM7qGMbvA/dZ4vIfWLAgAJ</a>), So I deployed with old version of jmx exporter from 0.6 to 0.9, still no <code>jvm_</code> metrics.</p>

<p>Are there anything I missed?</p>

<h3>env:</h3>

<p>kuberentes: AWS EKS (kubernetes version is 1.10.x)</p>

<p>public grafana dashboard: <a href="https://grafana.com/dashboards/721" rel="nofollow noreferrer">kafka overview</a></p>

<p>Just realised the owner of <code>jmx-exporter</code> mentioned in README:</p>

<blockquote>
  <p>This exporter is <code>intended to be run as a Java Agent</code>, exposing a HTTP server and serving metrics of the local JVM. It can be also run as an independent HTTP server and scrape remote JMX targets, <code>but this has various disadvantages</code>, such as being harder to configure and being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.</p>
</blockquote>

<p>Not really understood what's that meaning, until I saw this comment: </p>

<p><a href="https://github.com/prometheus/jmx_exporter/issues/111#issuecomment-341983150" rel="nofollow noreferrer">https://github.com/prometheus/jmx_exporter/issues/111#issuecomment-341983150</a></p>

<blockquote>
  <p>@brian-brazil can you add some sort of tip to the readme that jvm_* metrics are only exposed when using the Java agent? It took me an hour or two of troubleshooting and searching old issues to figure this out, after playing only with the HTTP server version. Thanks!</p>
</blockquote>

<p>So jmx-exporter has to be run with <code>java agent</code> to get <code>jvm_</code> metric. <code>jmx_prometheus_httpserver</code> doesn't support, but it is the default setting in kafka helm chart.</p>

<p><a href="https://github.com/kubernetes/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L82" rel="nofollow noreferrer">https://github.com/kubernetes/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L82</a></p>

<pre><code>command:
- sh
- -exc
- |
  trap "exit 0" TERM; \
  while :; do \
  java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 \
  -XshowSettings:vm \
  -jar \
  jmx_prometheus_httpserver.jar \              # &lt;&lt;&lt; here
  {{ .Values.prometheus.jmx.port | quote }} \
  /etc/jmx-kafka/jmx-kafka-prometheus.yml &amp; \
  wait $! || sleep 3; \
  done
</code></pre>
