<p>I'm looking to capture logs from a pod in kubernetes for two use cases:</p>
<ol>
<li>Realtime -&gt; for which I'm using <code>kubectl logs ---</code> right now</li>
<li>Not realtime -&gt; using <code>stackdriver</code> to pipe to <code>bigquery</code></li>
</ol>
<p>For both use cases, everything is working, however, when a container exits early due to an error, I lose the logs (i.e. stackdriver doesn't pick them up fast enough).</p>
<p>Is this latency documented somewhere? And assuming stackdriver isn't fast enough, is there another logging solution that would prove more effective? I'm considering having a sidecar container that captures logs, but I'm not sure if this is the best approach.</p>

<p>The logging stack on GKE uses <a href="https://cloud.google.com/stackdriver/docs/solutions/gke/managing-logs#log_collection_with_custom_fluentd" rel="nofollow noreferrer">fluentd</a> to pick the logs from the <code>stdout</code>, <code>stderr</code> that the <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#using-a-node-logging-agent" rel="nofollow noreferrer">container runtime writes to the nodes, as show in the <code>node logging agent</code> approach</a>.</p>
<p>This isn't <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level" rel="nofollow noreferrer">much different</a> from what you do when you use <code>kubectl logs</code>:</p>
<blockquote>
<p>When you run <code>kubectl logs</code> as in the basic logging example, the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response.</p>
</blockquote>
<p>You issue <strong>doesn't sound like Stackdriver isn't fast enough</strong> but, your container runtime is, for some reason not writing logs to the aforementioned log file where fluentd picks the logs before exporting them.</p>
<p>Before changing the logging architecture, you might want to <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/" rel="nofollow noreferrer">determine the reasons for pod failure</a> and even <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#customizing-the-termination-message" rel="nofollow noreferrer">customize the termination message path</a> in order to later retrieve it with a <a href="https://cloud.google.com/stackdriver/docs/solutions/gke/managing-logs#log_collection_with_custom_fluentd" rel="nofollow noreferrer">custom fluentd log collector</a>.</p>
<p>If this doesn't suit your needs, you can try <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/" rel="nofollow noreferrer">Elasticsearch</a> instead.</p>
<p>As for the sidecar approach, while it's completely feasible, the <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent" rel="nofollow noreferrer">official documentation</a> warns on some drawback on this approach:</p>
<blockquote>
<p>Using a logging agent in a sidecar container can lead to significant resource consumption. Moreover, you won't be able to access those logs using kubectl logs command, because they are not controlled by the kubelet.</p>
</blockquote>
<p>Finally, you should also consider that all the previous information relies on the fact that the <strong>container gets to the phase of creation and it's able to write to the log file</strong>. If your containers are having &quot;<em>early exits</em>&quot;, meaning that <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase" rel="nofollow noreferrer">aren't even created</a>, then the logs might not even be there for a start, and Stackdriver will never pick them.</p>
<p><em>Edit:</em></p>
<p>To mention that you want to also consider that a failed container needs to write to both outputs, <code>stdout</code> and <code>stderr</code>. If it's failing &quot;silently&quot;, that will also won't be reflected in Stackdriver.</p>
