<p>I am experimenting jenkins in k8s cluster. My environment is minikube.</p>
<p>I setup a standalone jenkins server in ubuntu, then I used <a href="https://plugins.jenkins.io/kubernetes/" rel="nofollow noreferrer">k8s plugin</a> to startup slave pods for jobs. Sometimes when I misconfigured, pods are very short-lived. Those pods exist only seconds, there are logs, but they disappear when pods are gone.</p>
<p>I tried <a href="https://grafana.com/oss/loki/" rel="nofollow noreferrer">loki grafana</a> to collect logs for analysis. I installed loki in k8s cluster using <a href="https://github.com/grafana/helm-charts/tree/main/charts/loki-stack" rel="nofollow noreferrer">loki-stack</a>. With some tweaking, loki-grafana works. I can see logs of most of the pods in grafana except those started by jenkins.</p>
<p>My question is, is it possible to collect logs of those short-lived pods? Is there anything I need to configure? Or is it simply impossible?</p>

<p><a href="https://grafana.com/docs/loki/latest/clients/promtail/configuration/#target_config" rel="nofollow noreferrer">https://grafana.com/docs/loki/latest/clients/promtail/configuration/#target_config</a></p>
<pre><code># Period to resync directories being watched and files being tailed to discover
# new ones or stop watching removed ones.
sync_period: &quot;10s&quot;
</code></pre>
<p>Promtail will (re-)discover new log files every 10s.</p>
<p>If you can make your Jenkins pods to live a little longer than 10s, then their logs will be discovered by and tailed by Promtail.</p>
<p>For example, you can attach a preStop handler, which sleep 10s...</p>
<p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/</a></p>
