<p>We have setup Prometheus + Grafana on our GKE cluster using the <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">stable/prometheus-operator</a> helm chart. Now we want to export some metrics to Stackdriver because we have installed custom metrics Stackdriver adapter. We are already using some Pub/Sub metrics from Stackdriver for autoscaling few deployments. Now we also want to use some Prometheus metrics (mainly nginx request rate) in the autoscaling of other deployments.</p>

<p>So, my first question: Can we use Prometheus adapter in parallel with Stackdriver adapter for autoscaling in the same cluster?</p>

<p>If not, we will need to install Stackdriver Prometheus Sidecar for exporting the Prometheus metrics to Stackdriver and then use them for autoscaling via Stackdriver adapter.</p>

<p>From the instructions <a href="https://cloud.google.com/stackdriver/docs/solutions/kubernetes-engine/prometheus#configuration" rel="nofollow noreferrer">here</a>, it looks like we need to install Stackdriver sidecar on same pod on which Prometheus is running. I gave it a try. When I run the <code>patch.sh</code> script, I got the message back: <code>statefulset.apps/prometheus-prom-operator-prometheus-o-prometheus patched</code> but when I inspected the statefulset again, it didn't have the Stackdriver sidecar container in it. Since this statefulset is created by a Helm chart, we probably can't modify it directly. Is there a recommended way of doing this in Helm?</p>

<p>Thanks to <a href="https://github.com/coreos/prometheus-operator/issues/2607#issuecomment-493104314" rel="nofollow noreferrer">this comment on GitHub</a>, I figured it out. There are so many configuration options accepted by this Helm chart that I missed it while reading the docs.</p>
<p>So, turns out that this Helm chart accepts a configuration option <code>prometheus.prometheusSpec.containers</code>. Its description in the <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">docs</a> says: &quot;Containers allows injecting additional containers. This is meant to allow adding an authentication proxy to a Prometheus pod&quot;. But obviously, it is not limited to the authentication proxy and you can pass any container spec here and it will be added to Prometheus StatefulSet created by this Helm chart.</p>
<p>Here is the sample configuration I used. Some key points:</p>
<ol>
<li>Please replace the values in angle brackets with your actual values.</li>
<li>Feel free to remove the arg <code>--include</code>. I added it because <code>nginx_http_requests_total</code> is the only Prometheus metric I want to send to Stackdriver for now. Check <a href="https://cloud.google.com/stackdriver/docs/solutions/kubernetes-engine/prometheus#managing-costs" rel="nofollow noreferrer">Managing costs for Prometheus-derived metrics</a> for more details about it.</li>
<li>To figure out the name of volume to use in <code>volumeMounts</code>:
<ol>
<li>List down StatefulSets in Prometheus Operator namespace. Assuming that you installed it in <code>monitoring</code> namespace:  <code>kubectl get statefulsets -n monitoring</code></li>
<li>Describe the Prometheus StatefulSet assuming that its name is <code>prometheus-prom-operator-prometheus-o-prometheus</code>: <code>kubectl describe statefulset prometheus-prom-operator-prometheus-o-prometheus -n monitoring</code></li>
<li>In details of this StatefulSet, find container named <code>prometheus</code>. Note the value passed to it in arg <code>--storage.tsdb.path</code></li>
<li>Find the volume that is mounted on this container on same path. In my case, it was <code>prometheus-prom-operator-prometheus-o-prometheus-db</code> so I mounted the same volume on my Stackdriver sidecar container as well.</li>
</ol>
</li>
</ol>
<pre><code>prometheus:
  prometheusSpec:
    containers:
      - name: stackdriver-sidecar
        image: gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:0.7.5
        imagePullPolicy: Always
        args:
          - --stackdriver.project-id=&lt;GCP PROJECT ID&gt;
          - --prometheus.wal-directory=/prometheus/wal
          - --stackdriver.kubernetes.location=&lt;GCP PROJECT REGION&gt;
          - --stackdriver.kubernetes.cluster-name=&lt;GKE CLUSTER NAME&gt;
          - --include=nginx_http_requests_total
        ports:
          - name: stackdriver
            containerPort: 9091
        volumeMounts:
          - name: prometheus-prom-operator-prometheus-o-prometheus-db
            mountPath: /prometheus
</code></pre>
<p>Save this yaml to a file. Let's assume you saved it to <code>prom-config.yaml</code></p>
<p>Now, find the release name you have used to install Prometheus Operator Helm chart on your cluster:</p>
<pre><code>helm list
</code></pre>
<p>Assuming that release name is <code>prom-operator</code>, you can update this release according to the config composed above by running this command:</p>
<pre><code>helm upgrade -f prom-config.yaml prom-operator stable/prometheus-operator
</code></pre>
<p>I hope you found this helpful.</p>
