<p>We are running java based microservices and we have following scenario </p>

<ul>
<li>Application writes debug.log file to /opt/tomcat/logs/debuglog/debug.log and size of log file is 1GB/hour </li>
<li>Tomcat to write catalina.out , localhost_access_log and localhost.log  and size of log is 1GB/hour for all of them</li>
</ul>

<p>Question is how to solve this problem when we have a lot of logs to centralised and analysed. We are running 20 instances of this application. We have got 150GB logs in flat filess. Followings are the problems, </p>

<ul>
<li>Store the logs for 3 years in GCS as per our SLA</li>
<li>Parse these logs and store them in BQ for bigdata for 1 year</li>
<li>Parse these logs and store them in ELK for 7 days for developers to analyse any running issue</li>
</ul>

<p>We are trying to evaluate following, </p>

<ul>
<li>As kubernetes recommends to run sidecars for application logs, we may endup running 3 sidecars considering catalina.out will go to stdout. We can use Stack-driver to process the logs and put them to GCS. Problem we see is container explosion specifically with auto scaling. Other problem is to parse the logs from stackdriver to BigQuery or ELK.</li>
<li>Mount GCS in containers and write there itself. Problem is the GCS is community driven and not production ready. We still have to write solution to parse these logs again</li>
<li>Use external drive mount to Minion and volume mount to container. Run 1 container per VM to process the logs for different pipelines and scenarios. This is solving a few problems for us like : Logs will not be lost when downscaled, No container explosion , single responsibly container to process different pipelines, move the logs to GCS as per availability. Problem we see is to manage the SSD storage attached to each VM upon up-scale and down-scale. </li>
</ul>

<p>Any suggestions are welcomed. </p>

<h2>EDIT</h2>

<p>We end up using custom pipeline on GCP where applications are pushing logs to pub/sub and dataflow is responsible to aggregate and transform the information.  </p>

<p>You can use a single sidecar that runs something like <a href="http://www.fluentd.org/" rel="nofollow noreferrer">fluentd</a> or <a href="https://www.elastic.co/products/logstash" rel="nofollow noreferrer">logstash</a>.
Both are log ingestion tools that can be customized with several plugins, which allow you to route to all destinations at once.
In the case of logstash you might even want to use filebeat.</p>

<p>Also, fluentd seems to have an <a href="http://www.fluentd.org/plugins#google-cloud-platform" rel="nofollow noreferrer">official plugin from Google that does most of what you want</a>.</p>

<h1>Using DaemonSets to collect logs on hosts</h1>

<p>This is the procedure described <a href="http://blog.kubernetes.io/2015/06/cluster-level-logging-with-kubernetes.html" rel="nofollow noreferrer">in this k8s blog post about cluster-level logging</a> and <a href="http://docs.fluentd.org/v0.12/articles/kubernetes-fluentd" rel="nofollow noreferrer">this blog post in the fluentd blog</a>.</p>

<p>The idea is to run a DaemonSet (a set of pods that runs on every node in the cluster) that mounts the host path where container log files are located.</p>

<p>However, this will only collect the logs that your application produces to stdout.
To collect the other ones, you can use the technique described <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#streaming-sidecar-container" rel="nofollow noreferrer">here</a>: run an <em>extremely lightweight</em> sidecar that just tails the log files.</p>
