<p>I'm following the  <a href="http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/" rel="nofollow noreferrer">k8s logging instructions</a> on how to configure cluster level logging. I'm using <a href="https://github.com/coreos/coreos-kubernetes/releases" rel="nofollow noreferrer">kube-aws cli Tool</a> to configure the cluster, and I can't seem to find a way to make it work.
I've tried setting the env vars as they mentioned in the k8s logging guide (KUBE_ENABLE_NODE_LOGGING and KUBE_LOGGING_DESTINATION) before running <code>kube-aws up</code> but that didn't seem to change anything.</p>
<p>After that, I've tried running the es and kibana rc's and services manually by taking them from the <strong>cluster/addons/fluentd-elasticsearch</strong> directory on k8s github repo, but that ran only those specific services and not the <strong>fluentd-elasticsearch</strong> service which supposed to run also by the tutorial example.</p>
<p>running <code>kubectl get pods --namespace=kube-system</code> returns the following:</p>
<p><a href="https://i.stack.imgur.com/acR1k.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/acR1k.png" alt="enter image description here" /></a></p>
<p>where we can see that the <code>fluentd-elasticsearch-kubernetes-node</code> is missing.</p>
<p>Also tried connecting to the cluster but failed with:</p>
<blockquote>
<p>unauthorized</p>
</blockquote>
<p>following the <a href="http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/" rel="nofollow noreferrer">k8s logging instructions</a> and running the command <code>kubectl config view</code> didn't return any username and password, and when tried accessing the es url, I didn't get any dialog with asking for username and password. Not sure if it related to the first issue.</p>
<p>Not sure what I'm missing here.</p>
<p>Thanks.</p>

<p>I've managed to get the cluster-level logging running on a small testing cluster started through the CoreOS <code>kube-aws</code> tool using the following steps. Please be aware that although I've had this running, I haven't really played with it sufficiently to be able to guarantee that all works correctly!</p>

<p><em>Enable log collection on nodes</em></p>

<p>You'll need to edit the <code>cloud-config-worker</code> and <code>cloud-config-controller</code> to export kubelet-collected logs and create the log directory </p>

<p><code>[Service]
Environment="RKT_OPTS=--volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log"
Environment=KUBELET_VERSION=v1.2.4_coreos.1
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStart=/usr/lib/coreos/kubelet-wrapper \
  --api-servers=http://127.0.0.1:8080 \
  --config=/etc/kubernetes/manifests
  ...other flags...</code></p>

<p>(taken from the 'Use the cluster logging add-on' section <a href="https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html" rel="nofollow">here</a>)</p>

<p><em>Install the logging components</em>
I used the components from <a href="https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch" rel="nofollow">here</a> (as you've already attempted). As you  noticed, this does not run fluentd, and assumes that it is run as part of the cluster bootstrapping. To get fluentd running I've extracted the fluentd Daemonset definition discussed <a href="https://github.com/coreos/coreos-kubernetes/issues/320" rel="nofollow">here</a> into a separate file:</p>

<p><code>{
  "apiVersion": "extensions\/v1beta1",
  "kind": "DaemonSet",
  "metadata": {
    "name": "fluent-elasticsearch",
    "namespace": "kube-system",
    "labels": {
      "k8s-app": "fluentd-logging"
    }
  },
  "spec": {
    "template": {
      "metadata": {
        "name": "fluentd-elasticsearch",
        "namespace": "kube-system",
        "labels": {
          "k8s-app": "fluentd-logging"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "fluentd-elasticsearch",
            "image": "gcr.io\/google_containers\/fluentd-elasticsearch:1.15",
            "resources": {
              "limits": {
                "memory": "200Mi"
              },
              "requests": {
                "cpu": "100m",
                "memory": "200Mi"
              }
            },
            "volumeMounts": [
              {
                "name": "varlog",
                "mountPath": "\/var\/log"
              },
              {
                "name": "varlibdockercontainers",
                "mountPath": "\/var\/lib\/docker\/containers",
                "readOnly": true
              }
            ]
          }
        ],
        "terminationGracePeriodSeconds": 30,
        "volumes": [
          {
            "name": "varlog",
            "hostPath": {
              "path": "\/var\/log"
            }
          },
          {
            "name": "varlibdockercontainers",
            "hostPath": {
              "path": "\/var\/lib\/docker\/containers"
            }
          }
        ]
      }
    }
  }
}</code></p>

<p>this Daemonset runs fluentd on each of the cluster nodes. </p>

<p>(NOTE: Whilst I've only tried adding these components after the cluster is already running, there's no reason you shouldn't be able to add these to to the <code>cloud-config-controller</code> in order to bring these up at the same time the cluster is started - which is more inline with that discussed on the referenced <a href="https://github.com/coreos/coreos-kubernetes/issues/320" rel="nofollow">issue</a>)</p>

<p>These instruction all assume that you're working with a cluster that you're happy to restart, or haven't yet started, in order to get the logging running - which I assume from your question is the situation you're in. I've also been able to get this working on a pre-existing cluster, by manually editing the AWS settings, and can add additional information on doing this if that is in fact what you are trying to do.</p>
