<p>My objective is to fetch the time series of a metric for a pod running on a kubernetes cluster on GKE using the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeSeries" rel="nofollow noreferrer">Stackdriver TimeSeries REST API</a>.</p>

<p>I have ensured that Stackdriver monitoring and logging are enabled on the kubernetes cluster.</p>

<p>Currently, I am able to fetch the time series of all the resources available <strong>in a cluster</strong> using the following filter:</p>

<pre><code>metric.type="container.googleapis.com/container/cpu/usage_time" AND resource.labels.cluster_name="&lt;MY_CLUSTER_NAME&gt;"
</code></pre>

<p>In order to fetch the time series of a <strong>given pod id</strong>, I am using the following filter:</p>

<pre><code>metric.type="container.googleapis.com/container/cpu/usage_time" AND resource.labels.cluster_name="&lt;MY_CLUSTER_NAME&gt;" AND resource.labels.pod_id="&lt;POD_ID&gt;"
</code></pre>

<p>This filter returns an <strong>HTTP 200 OK</strong> with an empty response body. I have found the pod ID from the <code>metadata.uid</code> field received in the response of the following kubectl command:</p>

<pre><code>kubectl get deploy -n default &lt;SERVICE_NAME&gt; -o yaml
</code></pre>

<p>However, when I use the Pod ID of a <strong>background container</strong> spawned by GKE/Stackdriver, I do get the time series values.</p>

<p>Since I am able to see Stackdriver metrics of my pod on the GKE UI, I believe I should also get the metric values using the REST API.</p>

<p>My doubts/questions are:</p>

<ol>
<li>Am I fetching the Pod ID of my pod correctly using kubectl?</li>
<li>Could there be some issue with my cluster setup/service deployment due to which I'm unable to fetch the metrics?</li>
<li>Is there some other way in which I can get the time series of my pod using the REST APIs?</li>
</ol>

<ol>
<li><p>I wouldn't rely on <code>kubectl get deploy</code> for pod ids. I would get them with something like <code>kubectl -n default get pods | grep &lt;prefix-for-your-pod&gt; | awk '{print $1}'</code></p></li>
<li><p>I don't think so, but the best way to find out is opening a support ticket with GCP if you have any doubts.</p></li>
<li><p>Not that I'm aware of, Stackdriver is the monitoring solution in GCP. Again, you can check with GCP support. There are other tools that you can use to get metrics from Kubernetes like <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a>. There are multiple guides on the web on how to set it up with Grafana on k8s. This is <a href="https://github.com/giantswarm/kubernetes-prometheus" rel="nofollow noreferrer">one</a> for example.</p></li>
</ol>

<p>Hope it helps!</p>
