<p>Basically, I had installed Prometheues-Grafana from the <a href="https://github.com/prometheus-operator/kube-prometheus" rel="noreferrer">kube-prometheus-stack</a> using the provided helm chart repo <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="noreferrer">prometheus-community</a></p>
<pre><code># helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
</code></pre>
<p>They are working fine.</p>
<p>But the problem I am facing now is integrating <strong>Thanos</strong> with this existing <strong>kube-prometheus-stack</strong>.</p>
<p>I installed thanos from the <a href="https://artifacthub.io/packages/helm/bitnami/thanos" rel="noreferrer">bitnami helm chart repo</a></p>
<pre><code># helm repo add bitnami https://charts.bitnami.com/bitnami
# helm install thanos bitnami/thanos
</code></pre>
<p>I can load the Thanos Query Frontend GUI, but no metrics showing there.</p>
<p><a href="https://i.stack.imgur.com/CDfVO.png" rel="noreferrer"><img src="https://i.stack.imgur.com/CDfVO.png" alt="thanos metrics" /></a>
<a href="https://i.stack.imgur.com/ZlttQ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ZlttQ.png" alt="thanos store" /></a></p>
<p>I am struggling now to get it worked properly. Is it because of Thanos from a completely different helm chart and Prometheus-operator-grafana stack from another helm chart ?.</p>
<p>My Kubernetes cluster on AWS has been created using Kops. And, I use Gitlab pipeline and helm to deploy apps to the cluster.</p>

vjwilson
<p>It's not enough to simply install them, you need to <strong>integrate</strong> <code>prometheus</code> with <code>thanos</code>.</p>
<p>Below I'll describe all steps you need to perform to get the result.</p>
<p>First short theory. The most common approach to integrate them is to use <code>thanos sidecar</code> container for <code>prometheus</code> pod. You can read more <a href="https://www.infracloud.io/blogs/prometheus-ha-thanos-sidecar-receiver/" rel="noreferrer">here</a>.</p>
<p><strong>How this is done:</strong></p>
<p>(considering that installation is clean, it can be easily deleted and reinstalled from the scratch).</p>
<ol>
<li>Get <code>thanos sidecar</code> added to the <code>prometheus</code> pod.</li>
</ol>
<p>Pull <code>kube-prometheus-stack</code> chart:</p>
<pre><code>$ helm pull prometheus-community/kube-prometheus-stack --untar
</code></pre>
<p>You will have a folder with a chart. You need to modify <code>values.yaml</code>, two parts to be precise:</p>
<pre><code># Enable thanosService
prometheus:
  thanosService:
    enabled: true # by default it's set to false

# Add spec for thanos sidecar
prometheus:
  prometheusSpec:
    thanos:
      image: &quot;quay.io/thanos/thanos:v0.24.0&quot;
      version: &quot;v0.24.0&quot;
</code></pre>
<p>Keep in mind, this feature is still experimental:</p>
<blockquote>
<pre><code>## This section is experimental, it may change significantly without deprecation notice in any release.
## This is experimental and may change significantly without backward compatibility in any release.
## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#thanosspec
</code></pre>
</blockquote>
<p>Once it's done, install the <code>prometheus</code> chart with edited <code>values.yaml</code>:</p>
<pre><code>$ helm install prometheus . -n prometheus --create-namespace # installed in prometheus namespace
</code></pre>
<p>And check that sidecar is deployed as well:</p>
<pre><code>$ kubectl get pods -n prometheus | grep prometheus-0
prometheus-prometheus-kube-prometheus-prometheus-0       3/3     Running   0          67s
</code></pre>
<p>It should be 3 containers running (by default it's 2). You can inspect it in more details with <code>kubectl describe</code> command.</p>
<ol start="2">
<li>Setup <code>thanos</code> chart and deploy it.</li>
</ol>
<p>Pull the <code>thanos</code> chart:</p>
<pre><code>$ helm pull bitnami/thanos --untar
</code></pre>
<p>Edit <code>values.yaml</code>:</p>
<pre><code>query:
  dnsDiscovery:
    enabled: true
    sidecarsService: &quot;prometheus-kube-prometheus-thanos-discovery&quot; # service which was created before
    sidecarsNamespace: &quot;prometheus&quot; # namespace where prometheus is deployed
</code></pre>
<p>Save and install this chart with edited <code>values.yaml</code>:</p>
<pre><code>$ helm install thanos . -n thanos --create-namespace
</code></pre>
<p>Check that it works:</p>
<pre><code>$ kubectl logs thanos-query-xxxxxxxxx-yyyyy -n thanos
</code></pre>
<p>We are interested in this line:</p>
<pre><code>level=info ts=2022-02-24T15:32:41.418475238Z caller=endpointset.go:349 component=endpointset msg=&quot;adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]&quot; address=10.44.1.213:10901 extLset=&quot;{prometheus=\&quot;prometheus/prometheus-kube-prometheus-prometheus\&quot;, prometheus_replica=\&quot;prometheus-prometheus-kube-prometheus-prometheus-0\&quot;}&quot;
</code></pre>
<ol start="3">
<li>Now go to the UI and see that metrics are available:</li>
</ol>
<p><a href="https://i.stack.imgur.com/Sii8l.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Sii8l.png" alt="enter image description here" /></a></p>
<p><strong>Good article to read:</strong></p>
<ul>
<li><a href="https://medium.com/nerd-for-tech/deep-dive-into-thanos-part-ii-8f48b8bba132" rel="noreferrer">Deep Dive into Thanos-Part II</a></li>
</ul>
