<p>I am trying to implement logging on my AWS/CoreOS/Kubernetes (1.1) setup, which I've set up without kube-up. So far I've installed fluentd as a static <a href="https://github.com/kubernetes/kubernetes/blob/release-1.1/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml" rel="nofollow">pod</a> on all nodes and the <a href="https://github.com/kubernetes/kubernetes/tree/release-1.1/cluster/addons/fluentd-elasticsearch" rel="nofollow">Fluentd-Elasticsearch addon</a> replication controllers and services. However, it doesn't yet work. Specifically, Kibana crashes like this:</p>

<pre><code>ELASTICSEARCH_URL=http://elasticsearch-logging.kube-system:9200
{"@timestamp":"2016-03-14T22:54:04.906Z","level":"error","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.&lt;anonymous&gt; (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
{"@timestamp":"2016-03-14T22:54:04.908Z","level":"fatal","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.&lt;anonymous&gt; (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
</code></pre>

<p>What should I do?</p>

<p>FWIW, Elasticsearch is connectable at <a href="http://elasticsearch-logging.kube-system:9200/" rel="nofollow">http://elasticsearch-logging.kube-system:9200/</a>, although it does return status 503. That could be the issue for all I know.</p>

<pre><code># curl http://elasticsearch-logging.kube-system:9200/
{
  "status" : 503,
  "name" : "Puppet Master",
  "cluster_name" : "kubernetes-logging",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}
</code></pre>

<h2>fluentd-es.yaml</h2>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.13
    resources:
      limits:
        cpu: 100m
    args:
    - -q
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  terminationGracePeriodSeconds: 30
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
</code></pre>

<h2>es-controller.yaml</h2>

<pre><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: elasticsearch-logging-v1
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 2
  selector:
    k8s-app: elasticsearch-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - image: gcr.io/google_containers/elasticsearch:1.7
        name: elasticsearch-logging         
        resources:
          limits:
            cpu: 100m
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: es-persistent-storage
          mountPath: /data
      volumes:
      - name: es-persistent-storage
        emptyDir: {}
</code></pre>

<h2>es-service.yaml</h2>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Elasticsearch"
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging
</code></pre>

<h2>kibana-controller.yaml</h2>

<pre><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: kibana-logging-v1
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kibana-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: kibana-logging
        image: gcr.io/google_containers/kibana:1.3
        resources:
          limits:
            cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging.kube-system:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
</code></pre>

<h2>kibana-service.yaml</h2>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Kibana"
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging
</code></pre>

<p>The 503 from Elasticsearch is likely the issue.  It should be returning a 200 if all is well. Your first step should be to look at the Elasticsearch logs.  You can do this with the <code>kubectl logs POD</code> command.  Your es-controller and es-service YAML appears to be correct.</p>

<p>One thing to note is that the fluentd-elasticsearch container you are using provides a configuration suitable for systems using syslog, whereas CoreOS uses systemd/journald.  That configuration will likely still provide you logs from your docker containers, as long as you are using the default json-file log driver, but will not provide system logs.  To get the system logs, you have to use other means, e.g., <a href="https://github.com/reevoo/fluent-plugin-systemd" rel="nofollow">https://github.com/reevoo/fluent-plugin-systemd</a> or <a href="https://github.com/mheese/journalbeat" rel="nofollow">https://github.com/mheese/journalbeat</a>.</p>
