<p>I deployed a ray cluster on Kubernetes using kuberay and I want to monitor the cluster using prometheus metrics. After reading ray document, I know that there is service discovery file is generated on the head node <code>/tmp/ray/prom_metrics_service_discovery.json</code>. Using the below Prometheus config, Prometheus will automatically update the addresses that it scrapes based on the contents of Ray’s service discovery file.</p>
<pre class="lang-yaml prettyprint-override"><code># Prometheus config file

# my global config
global:
  scrape_interval:     2s
  evaluation_interval: 2s

# Scrape from Ray.
scrape_configs:
- job_name: 'ray'
  file_sd_configs:
  - files:
    - '/tmp/ray/prom_metrics_service_discovery.json'
</code></pre>
<p>But since I am using Kubernetes, based on humble experience, I think the most convenient way to configure Prometheus to scape the ray metrics should be exposing metrics configuration on service annotations like this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: xxx
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/path: /metrics
    prometheus.io/port: &quot;8081&quot;
</code></pre>
<p>Is there any way to achieve this?</p>
<hr />
<p>Updated at 2022-11-10
I manually add an Kubernetes service to solve this problem.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  # labels:
  #   app.kubernetes.io/name: kuberay-metrics
  #   ray.io/cluster: {{ include &quot;ray-cluster.fullname&quot; . }}
  name: {{ include &quot;ray-cluster.fullname&quot; . }}-metrics-svc
  annotations:
    {{- if .Values.prometheus.enable }}
    prometheus.io/scrape: &quot;{{.Values.prometheus.enable }}&quot;
    prometheus.io/path: /metrics
    prometheus.io/port: &quot;8080&quot;
    {{- end }}
spec:
  ports:
  - name: metrics
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: kuberay
    ray.io/cluster: {{ include &quot;ray-cluster.fullname&quot; . }}
  type: ClusterIP

</code></pre>

kk17
<p>Re: Is there any way to achieve this?</p>
<p>-- I don't think so, not yet!
I think this would be great to discuss in a <a href="https://github.com/ray-project/kuberay/issues/new?assignees=&amp;labels=enhancement&amp;template=feature-request.yml&amp;title=%5BFeature%5D+" rel="nofollow noreferrer">KubeRay feature request GitHub issue</a>. Please feel free to post there!</p>
