<p>how to set up efk logging in aks cluster nodes?</p>
<p>Below are my spec files for efk logging in aks clusters.</p>
<pre><code># Elasticsearch.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: logging
spec:
  serviceName: logs-elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 500m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data-logging
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: &quot;es-cluster-0.logs-elasticsearch,es-cluster-1.logs-elasticsearch,es-cluster-2.logs-elasticsearch&quot;
          - name: cluster.initial_master_nodes
            value: &quot;es-cluster-0,es-cluster-1,es-cluster-2&quot;
          - name: ES_JAVA_OPTS
            value: &quot;-Xms512m -Xmx512m&quot;
      initContainers:
      - name: fix-permissions
        image: busybox
        command: [&quot;sh&quot;, &quot;-c&quot;, &quot;chown -R 1000:1000 /usr/share/elasticsearch/data&quot;]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data-logging
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: [&quot;sysctl&quot;, &quot;-w&quot;, &quot;vm.max_map_count=262144&quot;]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: [&quot;sh&quot;, &quot;-c&quot;, &quot;ulimit -n 65536&quot;]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data-logging
      labels:
        app: elasticsearch
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      storageClassName: &quot;managed-premium-retain-sc&quot;
      resources:
        requests:
          storage: 100Gi

---
kind: Service
apiVersion: v1
metadata:
  name: logs-elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

########################
# Kibana yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.5.0
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 500m
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://logs-elasticsearch.logging.svc.cluster.local:9200
        ports:
        - containerPort: 5601

---
apiVersion: v1
kind: Service
metadata:
  name: logs-kibana
spec:
  selector: 
    app: kibana
  type: ClusterIP
  ports:
    - port: 5601
      targetPort: 5601

##################
# fluentd daemonset and rbac,sa,clusterrole specs

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  labels:
    app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  labels:
    app: fluentd
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: default

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: &quot;logs-elasticsearch.logging.svc.cluster.local&quot;
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: &quot;9200&quot;
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: &quot;http&quot;
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
          - name: FLUENT_UID
            value: &quot;0&quot;
        resources:
          limits:
            memory: 512Mi
            cpu: 500m
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log/
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers

          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log/
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers



</code></pre>
<p>The setup is working fine only thing no logs are coming to elasticsearch cluster from fluentd whereas the same spec files work fine inside minikube cluster.</p>
<p>As for this setup kibana is up and able to connect with elasticsearch and the same is the case with fluentd, just logs are not coming inside elasticseach.</p>
<p>What extra configuration needs to be configured to make these config files work with azure k8 service(AKS) cluster nodes?</p>

devops-admin
<p>Had to add below environment variables for Fluentd.</p>
<p>Reference Link: <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/issues/434" rel="nofollow noreferrer">https://github.com/fluent/fluentd-kubernetes-daemonset/issues/434</a></p>
<pre><code>          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: /var/log/containers/fluent*
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: /^(?&lt;time&gt;.+) (?&lt;stream&gt;stdout|stderr)( (?&lt;logtag&gt;.))? (?&lt;log&gt;.*)$/


</code></pre>
<p>Here's the complete spec.</p>
<pre><code># Elasticsearch.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: logging
spec:
  serviceName: logs-elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 500m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data-logging
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: &quot;es-cluster-0.logs-elasticsearch,es-cluster-1.logs-elasticsearch,es-cluster-2.logs-elasticsearch&quot;
          - name: cluster.initial_master_nodes
            value: &quot;es-cluster-0,es-cluster-1,es-cluster-2&quot;
          - name: ES_JAVA_OPTS
            value: &quot;-Xms512m -Xmx512m&quot;
      initContainers:
      - name: fix-permissions
        image: busybox
        command: [&quot;sh&quot;, &quot;-c&quot;, &quot;chown -R 1000:1000 /usr/share/elasticsearch/data&quot;]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data-logging
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: [&quot;sysctl&quot;, &quot;-w&quot;, &quot;vm.max_map_count=262144&quot;]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: [&quot;sh&quot;, &quot;-c&quot;, &quot;ulimit -n 65536&quot;]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data-logging
      labels:
        app: elasticsearch
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      storageClassName: &quot;managed-premium-retain-sc&quot;
      resources:
        requests:
          storage: 100Gi

---
kind: Service
apiVersion: v1
metadata:
  name: logs-elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

########################
# Kibana yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.5.0
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 500m
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://logs-elasticsearch.logging.svc.cluster.local:9200
        ports:
        - containerPort: 5601

---
apiVersion: v1
kind: Service
metadata:
  name: logs-kibana
spec:
  selector: 
    app: kibana
  type: ClusterIP
  ports:
    - port: 5601
      targetPort: 5601

##################
# fluentd daemonset and rbac,sa,clusterrole specs

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  labels:
    app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  labels:
    app: fluentd
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: default

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: &quot;logs-elasticsearch.logging.svc.cluster.local&quot;
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: &quot;9200&quot;
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: &quot;http&quot;
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
          - name: FLUENT_UID
            value: &quot;0&quot;
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: /var/log/containers/fluent*
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: /^(?&lt;time&gt;.+) (?&lt;stream&gt;stdout|stderr)( (?&lt;logtag&gt;.))? (?&lt;log&gt;.*)$/

        resources:
          limits:
            memory: 512Mi
            cpu: 500m
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log/
#        - name: varlibdockercontainers
#          mountPath: /var/lib/docker/containers
        - name: dockercontainerlogsdirectory
          mountPath: /var/log/pods
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log/
 #     - name: varlibdockercontainers
 #       hostPath:
#          path: /var/lib/docker/containers
      - name: dockercontainerlogsdirectory
        hostPath:
          path: /var/log/pods

</code></pre>
