<p>I've read that by default, clusters on GKE write all logs to stackdriver. On the GKE UI, I can see the container logs of all my deployments, but am looking to get these logs from an API programmatically.</p>

<p>I've tried to do something like <code>gcloud logging read "resource.labels.pod_id="&lt;pod-id&gt;""</code> but nothing is returned.</p>

<p>Do I need to do something special in order to enable logging on the cluster? And if not, how do I access logs for a specific deployment/pod/container? </p>

<p>Further, are these logs persistent? As in, when the deployment dies, can I still access these logs?</p>

<p>You have several options to read the <code>GKE</code> pods logs from Stackdriver. Some of them are: </p>

<blockquote>
  <h2>Reading logs</h2>
  
  <p>To read log entries in Logging, you can do any of the following:</p>
  
  <ul>
  <li>Use the Logs Viewer in the Google Cloud Console.</li>
  <li>Call the Logging API through the Client Libraries for your programming language.</li>
  <li>Call the Logging API REST endpoints directly. See the Logging API reference documentation.</li>
  <li><p>Use the Cloud SDK. For more information, see the gcloud logging command-line interface.</p>
  
  <p> <em><a href="https://cloud.google.com/logging/docs/setup" rel="nofollow noreferrer">Cloud.google.com: Logging: Setup </a></em> </p></li>
  </ul>
</blockquote>

<p>As for:</p>

<blockquote>
  <p>Do I need to do something special in order to enable logging on the cluster? And if not, how do I access logs for a specific deployment/pod/container?</p>
</blockquote>

<p>Please refer to: <a href="https://cloud.google.com/logging/docs/access-control" rel="nofollow noreferrer">Cloud.google.com: Logging: Access control</a></p>

<p>Answering below: </p>

<blockquote>
  <p>Further, are these logs persistent? As in, when the deployment dies, can I still access these logs?</p>
</blockquote>

<p>Yes you can still access this logs, even if the deployment was deleted. You can still access the logs even if you delete your cluster. Logs stored in Stackdriver have <strong>retention</strong> policies which will store the logs for set amount of time. 
Please refer to: </p>

<ul>
<li><a href="https://cloud.google.com/logging/quotas" rel="nofollow noreferrer">Cloud.google.com: Logging: Quotas</a></li>
<li><a href="https://cloud.google.com/logging/docs/storage" rel="nofollow noreferrer">Cloud.google.com: Logging: Storage</a></li>
</ul>

<hr>

<p>Please take a look on the example below which shows how to access logs with <code>gcloud</code> command:</p>

<p><strong>Steps:</strong> </p>

<ul>
<li>Create a <code>Deployment</code> which will send logs to Stackdriver</li>
<li>Check if the logs are stored in Stackdriver</li>
<li>Get the logs from Stackdriver with <code>gcloud</code></li>
</ul>

<h2>Create a Deployment</h2>

<p>Please follow a Google Cloud Platform guide to spawn a <code>Deployment</code> which will send data to Stackdriver: </p>

<ul>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling#before-you-begin" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Custom Metrics autoscaling: Before you begin</a> - start  </li>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling#exporting_metrics_from_the_application" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Custom Metrics autoscaling: Exporting</a> - end </li>
</ul>

<h2>Check if the logs are in Stackdriver.</h2>

<p>Logs exported by above deployment will be stored in Stackdriver. The example of it should look like this: </p>

<pre><code>{
 insertId: "REDACTED"  
 labels: {
  k8s-pod/pod-template-hash: "545464fb5"   
  k8s-pod/run: "custom-metric-sd"   
 }
 logName: "projects/REDACTED/logs/stderr"  
 receiveTimestamp: "2020-05-26T10:17:16.161949129Z"  
 resource: {
  labels: {
   cluster_name: "gke-logs"    
   container_name: "sd-dummy-exporter"    
   location: "ZONE"    
   namespace_name: "default"    
   pod_name: "custom-metric-sd-545464fb5-2rdvx"    
   project_id: "REDACTED"    
  }
  type: "k8s_container"   
 }
 severity: "ERROR"  
 textPayload: "2020/05/26 10:17:10 Finished writing time series with value: 0xc420015290
"  
 timestamp: "2020-05-26T10:17:10.356684667Z"  
}
</code></pre>

<p>Above log entry will help with creating a <code>gcloud</code> command to get only specified logs from the Stackdriver. </p>

<h2>Get the logs from Stackdriver with <code>gcloud</code></h2>

<p>As you pointed: </p>

<blockquote>
  <p>I've tried to do something like gcloud logging read "resource.labels.pod_id=""" but nothing is returned.</p>
</blockquote>

<p>The fact that nothing is returned is most probably connected with the fact that the requested resource was not found with the command you used. </p>

<p>To get this logs from Stackdriver invoke below command: </p>

<pre class="lang-sh prettyprint-override"><code>$ gcloud logging read "resource.type=k8s_container AND resource.labels.container_name=sd-dummy-exporter" 
</code></pre>

<p>Dividing above command on smaller pieces: </p>

<ul>
<li><code>resource.type=k8s_container</code> - it will get the logs with a type of <code>k8s_container</code> </li>
<li><code>resource.labels.container_name=XYZ</code> - it will get the logs with a specified <code>container_name</code>. </li>
</ul>

<p>This pieces are directly connected with example singular log entry mentioned earlier. </p>

<p>A tip: </p>

<blockquote>
  <p><code>resources.labels.container_name</code> can be used to collect logs from multiple pods as the specific pod can be referenced with <code>pod_name</code>. </p>
</blockquote>
