<p>I've installed <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">Prometheus operator</a> <code>0.34</code> (which works as expected) on cluster <strong>A</strong> (main prom)
Now I want to use the <a href="https://prometheus.io/docs/prometheus/latest/federation/" rel="nofollow noreferrer">federation</a> option,I mean collect metrics from other Prometheus which is located on other  K8S cluster <strong>B</strong> </p>

<p><strong>Secnario:</strong></p>

<blockquote>
  <ol>
  <li>have in cluster <strong>A</strong> MAIN <a href="https://github.com/coreos/prometheus-operator" rel="nofollow noreferrer">prometheus operator</a> <code>v0.34</code> <a href="https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml" rel="nofollow noreferrer">config</a></li>
  <li>I've in cluster <strong>B</strong> SLAVE <a href="https://github.com/helm/charts/tree/master/stable/prometheus" rel="nofollow noreferrer">prometheus</a> <code>2.13.1</code> <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml" rel="nofollow noreferrer">config</a></li>
  </ol>
</blockquote>

<p>Both installed successfully via helm, I can access to localhost via <code>port-forwarding</code> and see the scraping results on each cluster.</p>

<p><strong>I did the following steps</strong></p>

<p>Use on the operator (main cluster A)  <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/additional-scrape-config.md" rel="nofollow noreferrer">additionalScrapeconfig</a> 
I've added the following to the <code>values.yaml</code> file and update it via helm.</p>

<pre><code>additionalScrapeConfigs:
 - job_name: 'federate'  
   honor_labels: true
   metrics_path: /federate
   params:
     match[]:
       - '{job="prometheus"}'
       - '{__name__=~"job:.*"}'
   static_configs:
     - targets:
       - 101.62.201.122:9090 # The External-IP and port from the target prometheus on Cluster B
</code></pre>

<p>I took the target like following:</p>

<p>on prometheus inside <strong>cluster B</strong> (from which I want to collect the data) I use:</p>

<p><code>kubectl get svc -n monitoring</code></p>

<p>And get the following entries:</p>

<p>Took the <code>EXTERNAL-IP</code> and put it inside the <code>additionalScrapeConfigs</code> config entry.</p>

<p>Now I switch to cluster <code>A</code> and run <code>kubectl port-forward svc/mon-prometheus-operator-prometheus  9090:9090 -n monitoring</code> </p>

<p>Open the browser with <code>localhost:9090</code> see the graph's and click on <code>Status</code> and there Click on <code>Targets</code> </p>

<p>And see the new target with job <code>federate</code></p>

<p><a href="https://i.stack.imgur.com/8oPqd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8oPqd.png" alt="enter image description here"></a></p>

<p>Now my main question/gaps. (security &amp; verification) </p>

<ol>
<li>To be able to see that target <code>state</code> on green (see the pic) I configure the prometheus server in cluster <code>B</code> instead of using <code>type:NodePort</code> to use <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L835" rel="nofollow noreferrer"><code>type:LoadBalacer</code></a> which expose the metrics outside, this can be good for testing but I need to <strong>secure it</strong>, how it can be done ?
How to make the e2e works in <strong>secure way</strong>...</li>
</ol>

<p>tls
<a href="https://prometheus.io/docs/prometheus/1.8/configuration/configuration/#tls_config" rel="nofollow noreferrer">https://prometheus.io/docs/prometheus/1.8/configuration/configuration/#tls_config</a></p>

<p>Inside <strong>cluster A</strong> (main cluster) we use certificate for out services with <code>istio</code> like following which works</p>

<pre><code>tls:
   mode: SIMPLE
   privateKey: /etc/istio/oss-tls/tls.key
   serverCertificate: /etc/istio/oss-tls/tls.crt
</code></pre>

<p>I see that inside the doc there is an option to config</p>

<pre><code>    additionalScrapeConfigs:
     - job_name: 'federate'  
       honor_labels: true
       metrics_path: /federate
       params:
         match[]:
           - '{job="prometheus"}'
           - '{__name__=~"job:.*"}'
       static_configs:
         - targets:
           - 101.62.201.122:9090 # The External-IP and port from the target
#        tls_config:
#          ca_file: /opt/certificate-authority-data.pem
#          cert_file: /opt/client-certificate-data.pem
#          key_file: /sfp4/client-key-data.pem
#          insecure_skip_verify: true

</code></pre>

<p>But not sure which certificate I need to use inside the prometheus operator config , the certificate of the main prometheus A or the slave B?</p>

<ol>
<li>You should consider using <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/additional-scrape-config.md#additional-scrape-configuration" rel="nofollow noreferrer">Additional Scrape Configuration</a></li>
</ol>

<blockquote>
  <p><code>AdditionalScrapeConfigs</code> allows specifying a key of a Secret
  containing additional Prometheus scrape configurations. Scrape
  configurations specified are appended to the configurations generated
  by the Prometheus Operator.</p>
</blockquote>

<ol start="2">
<li><p>I am affraid this is not officially supported. However, you can update your <code>prometheus.yml</code> section within the Helm chart. If you want to learn more about it, check out <a href="https://www.promlts.com/resources/wheres-my-prometheus-yml?utm_source=sof&amp;utm_medium=organic&amp;utm_campaign=prometheus" rel="nofollow noreferrer">this blog</a></p></li>
<li><p>I see two options here:</p></li>
</ol>

<blockquote>
  <p>Connections to Prometheus and its exporters are not encrypted and
  authenticated by default. <a href="https://0x63.me/tls-between-prometheus-and-its-exporters/" rel="nofollow noreferrer">This is one way of fixing that with TLS
  certificates and
  stunnel</a>.</p>
</blockquote>

<p>Or specify <a href="https://prometheus.io/docs/operating/security/#secrets" rel="nofollow noreferrer">Secrets</a> which you can add to your scrape configuration.</p>

<p>Please let me know if that helped.  </p>
