<p>We're using <a href="https://github.com/fluent/fluentd-kubernetes-daemonset" rel="nofollow noreferrer">https://github.com/fluent/fluentd-kubernetes-daemonset</a> to deploy Fluentd in our K8s cluster. We have 5 nodes in the cluster, which means there are 5 Fluentd pods.</p>
<p>Each Fluentd pod in the DaemonSet exposes Prometheus metrics on <code>localhost:24231/metrics</code> endpoint via fluentd prometheus plugin. I'm having trouble finding the relevant bits of documentation on how to configure Prometheus to collect those metrics from every Pod's localhost:24321/metrics endpoint.</p>
<h2>TL;DR</h2>
<ul>
<li>there are N pods in a DaemonSet</li>
<li>each pod has prometheus metrics exposed on localhost:24321</li>
</ul>
<p>I need to configure Prometheus so it's able to scrape those metrics somehow. Any tips on how to solve this or examples of such configurations would be much appreciated!</p>

Roman Kolpak
<p>Solution in our case was using <a href="https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#include-podmonitors" rel="nofollow noreferrer">pod monitors</a> CRD from the prometheus operator with <code>podMetricEndpoints</code> pointing at the right port:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: fluentd
spec:
  podMetricsEndpoints:
    - honorLabels: true
      interval: 15s
      path: /metrics
      targetPort: 24231
      scheme: http
  selector:
    matchLabels:
      app: fluentd
</code></pre>
