<p>We are planning to implement Kubernetes application monitoring using Prometheus. Our application is running in apache and deployed to kubernetes cluster. We are developing Custom Apache exporter which will run in same POD as my application container &amp; gather metrics from apache and push them to Prometheus. As part of this monitoring we need to parse access.log in apache to get number of 2xx, 3xx,4xx &amp; 5xx requests. But Our application that is deployed in Kubernetes doesnt store any logs inside the POD, but it logs to STDOUT. So I can access logs from kubectl logs command. Now my Apache exporter that I am building is hosted in same POD as my application container. Now my question is How can i get application logs (access.log) to be present/accessible to  my apache exporter container, so that I can parse the logic and comeup with metrics like number of 2xx,3xx,4xx &amp; 5xx.</p>

Anurag Bhargav
<p>Kubernetes stores container's STDOUTs at <code>/var/logs/containers</code> and <code>/var/logs/pods</code> of each node in a format <code>POD_NAMESPACE_CONTAINER_CONTAINERID.log</code> and <code>NAMESPACE_POD_CONTAINERID/CONTAINER/*.log</code> respectively. These files will be deleted once the pod is evitced from the node.</p>

<p>You can mount either of them to your exporter pod to read logs. Problems are,</p>

<ul>
<li><code>CONTAINER ID</code> is not available as an ENV in Kubernetes. We can't generate the file name and mount exactly the file that we want. We have to mount the entire <code>/var/log/containers/</code> or <code>/var/log/pods</code> directory. </li>
<li>Mounting a writable <code>hostPath</code> is highly vulnerable in Kubernetes. So, make sure to set it as <code>readOnly</code> file system or configure <code>PodSecurityPolicy</code>.</li>
</ul>

<p>YAML</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    resources: {}
  - image: busybox
    name: exporter
    command: ["cat"]
    tty: true
    volumeMounts:
    - mountPath: /var/log/
      name: logs
      readOnly: true
  volumes:
  - name: logs
    hostPath:
      path: /var/log/containers/
      # path: /var/log/pods/
      type: Directory
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}
</code></pre>

<p>Logs</p>

<pre><code>$ kubectl exec -it nginx -n dev -c exporter sh -- ls /var/log | grep nginx
nginx_dev_exporter-41a2a80de1b899d1ee848378d2bf31285658bf86700671e560588ac69b2717b4.log
nginx_dev_nginx-e4c12df0c9dae129ed113f4195d750b447a570a686e3b5608e1af37ced788549.log
</code></pre>
