<p>I am trying to set up EFK stack on my k8s cluster using <a href="https://github.com/kubernetes/contrib/blob/master/ansible/roles/kubernetes-addons/tasks/cluster-logging.yml" rel="nofollow noreferrer">ansible repo</a>.</p>

<p>When i tried to browse kibana dashboard it shows me next output:</p>

<p><a href="https://i.stack.imgur.com/2P12B.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2P12B.png" alt="kibana dash"></a></p>

<p>After making some research, i found out that i don't have any log detected by Fluentd.
I am running k8s 1.2.4 on minions and 1.2.0 on master.
What i succeeded to understand, is that kubelet creates /var/log/containers directory, and make symlinks from all containers running in the cluster into it. After that Fluentd mounts share /var/log volume from the minion and have eventually access to all logs containers. So , it can send these logs to elastic search.</p>

<p>In my case i had /var/log/containers created, but it is empty, even /var/lib/docker/containers does not contain any log file.
I used to use the following controllers and services for EFK stack setup:</p>

<h1>es-controller.yaml</h1>

<p><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: elasticsearch-logging-v1
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 2
  selector:
    k8s-app: elasticsearch-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - image: gcr.io/google_containers/elasticsearch:v2.4.1
        name: elasticsearch-logging
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: es-persistent-storage
          mountPath: /data
        env:
        - name: "NAMESPACE"
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumes:
      - name: es-persistent-storage
        emptyDir: {}</code></p>

<h1>es-service.yaml</h1>

<p><code>apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Elasticsearch"
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging</code></p>

<h1>fluentd-es.yaml</h1>

<p><code>apiVersion: v1
kind: Pod
metadata:
  name: fluentd-es-v1.20
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    version: v1.20
spec:
  containers:
  - name: fluentd-es
    image: gcr.io/google_containers/fluentd-elasticsearch:1.20
    command:
      - '/bin/sh'
      - '-c'
      - '/usr/sbin/td-agent 2&gt;&amp;1 &gt;&gt; /var/log/fluentd.log'
    resources:
      limits:
        cpu: 100m
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  terminationGracePeriodSeconds: 30
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers</code></p>

<h1>kibana-controller.yaml</h1>

<p><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
    spec:
      containers:
      - name: kibana-logging
        image: gcr.io/google_containers/kibana:v4.6.1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP</code></p>

<h1>kibana-service.yaml</h1>

<p><code>apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Kibana"
spec:
  type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging</code></p>

<h1>update:</h1>

<p>I changed fluentd-es.yaml as following:</p>

<p><code>apiVersion: v1
kind: Pod
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.15
    resources:
      limits:
        memory: 200Mi
      requests:
        cpu: 100m
        memory: 200Mi
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  terminationGracePeriodSeconds: 30
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers</code></p>

<p>But when i run a pod "named gateway", i got in the fluentd log the next error:
 <code>/var/log/containers/gateway-c3cuu_default_gateway-d5966a86e7cb1519329272a0b900182be81f55524227db2f524e6e23cd75ba04.log unreadable. It is excluded and would be examined next time.</code></p>

<p>Finally i found out what was causing the issue.
when installing docker from CentOS 7 repo, there is an option (--log-driver=journald) which force docker to run log output to journald. The default behavior is to write these logs to json.log files.So, the only thing i had to do, delete the last mentioned option from /etc/sysconfig/docker.</p>
