<p>i have a rather recent kubernetes cluster running on GCE. I am trying to get my application to log to Cloud Logging / Stackdriver.</p>

<p>I can see all the kubernetes cluster logs there but no container output ever materializes.</p>

<p>So when I follow this guide: <a href="http://kubernetes.io/docs/getting-started-guides/logging/" rel="nofollow">http://kubernetes.io/docs/getting-started-guides/logging/</a>, I can see the output of the pod</p>

<pre><code>kubectl logs counter
2163: Wed Aug 31 15:02:52 UTC 2016
</code></pre>

<p>This never makes it to the Logging Interface</p>

<p><a href="http://i.stack.imgur.com/N7qlF.png" rel="nofollow">Pod not showing in selector</a></p>

<p>The fluentd-cloud-logging pods give no logging output</p>

<pre><code>kubectl logs --namespace=kube-system fluentd-cloud-logging-staging-minion-group-20hk
</code></pre>

<p>The /var/log/google-fluentd/google-fluentd.log file looks happy</p>

<pre><code>...
2016-08-31 14:07:16 +0000 [info]: following tail of /var/log/containers/node-problem-detector-v0.1-hgtcr_kube-system_POD-07e5b134c9f8ff48f73f1df41473a84a07738ac750840f09938d604694c4bd6e.log
2016-08-31 14:07:16 +0000 [info]: following tail of /var/log/containers/rails-2607986313-s7r5e_default_POD-9f1dd02f23de552a40297f761d09c03b50e5a2cd9789ef498139d24602d9847e.log
2016-08-31 14:07:16 +0000 [info]: following tail of /var/log/salt/minion
2016-08-31 14:07:16 +0000 [info]: following tail of /var/log/startupscript.log
2016-08-31 14:07:16 +0000 [info]: following tail of /var/log/docker.log
2016-08-31 14:07:16 +0000 [info]: following tail of /var/log/kubelet.log
2016-08-31 14:07:22 +0000 [info]: Successfully sent to Google Cloud Logging API.
2016-08-31 14:07:22 +0000 [info]: Successfully sent to Google Cloud Logging API.
</code></pre>

<p>Kubernetes Version is</p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:29:08Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>Cluster was started with</p>

<pre><code>export KUBE_GCE_ZONE=europe-west1-d
export NODE_SIZE=n1-standard-2
export NUM_NODES=2
export KUBE_GCE_INSTANCE_PREFIX=staging
export ENABLE_CLUSTER_AUTOSCALER=true
export KUBE_ENABLE_CLUSTER_MONITORING=true
export KUBE_ENABLE_CLUSTER_MONITORING=google
</code></pre>

<p>Any ideas what I might be doing wrong? To my understanding this should work out of the box, right?</p>

<p>Bit of a long shot, but have you enabled the logging API?</p>

<p>"You can do so from the Developers Console, <a href="https://console.developers.google.com/apis/api/logging/overview" rel="noreferrer">here</a>. Try going there, clicking the Enable API button, and seeing whether the errors keep coming."</p>

<p><a href="https://github.com/kubernetes/kubernetes/issues/20516" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/20516</a></p>

<p><a href="https://stackoverflow.com/questions/35165973/google-cloud-logging-google-fluentd-dropping-messages">Google Cloud Logging + google-fluentd Dropping Messages</a></p>
