<p>I am using the ELK stack (elasticsearch, logstash, kibana) for log processing and analysis in a Kubernetes environment. To capture logs I am using filebeat.</p>
<p>The service account, the cluster role, and the cluster role binding of elasticsearch below yaml</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    k8s-app: elasticsearch
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: elasticsearch
  labels:
    k8s-app: elasticsearch
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - &quot;services&quot;
  - &quot;namespaces&quot;
  - &quot;endpoints&quot;
  verbs:
  - &quot;get&quot;
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: elasticsearch
  labels:
    k8s-app: elasticsearch
subjects:
- kind: ServiceAccount
  name: elasticsearch
  namespace: kube-system
  apiGroup: &quot;&quot;
roleRef:
  kind: ClusterRole
  name: elasticsearch
  apiGroup: &quot;&quot;
</code></pre>
<p>elasticsearch service yaml</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    k8s-app: elasticsearch
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch
  externalIPs:
  - 10.10.0.82
</code></pre>
<p>Elastic search statesul set yaml below:</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    k8s-app: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 2
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      k8s-app: elasticsearch
  template:
    metadata:
      labels:
        k8s-app: elasticsearch
    spec:
      serviceAccountName: elasticsearch
      containers:
      - image: elasticsearch:6.8.4
        name: elasticsearch
        resources:
            limits:
              cpu: 1000m
              memory: &quot;2Gi&quot;
            requests:
              cpu: 100m
              memory: &quot;1Gi&quot;
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /data
        env:
        - name: &quot;NAMESPACE&quot;
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      initContainers:
      - image: alpine:3.6
        command: [&quot;/sbin/sysctl&quot;, &quot;-w&quot;, &quot;vm.max_map_count=262144&quot;]
        name: elasticsearch-init
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        k8s-app: elasticsearch
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      resources:
        requests:
          storage: 10Gi
</code></pre>
<p>pv &amp; pvc0 yaml</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: elklogs-pv0
  namespace: kube-system
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    server: 10.10.0.131
    path: /opt/data/vol/0
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: data-elasticsearch-0
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
</code></pre>
<p>pv_pvc1.yaml</p>
<pre><code>---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: elklogs-pv1
  namespace: kube-system
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    server: 10.10.0.131
    path: /opt/data/vol/1

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: data-elasticsearch-1
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
</code></pre>
<p>logstash_svc.yaml</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: logstash-service
  namespace: kube-system
spec:
  selector:
    app: logstash
  ports:
  - protocol: TCP
    port: 5044
    targetPort: 5044
  externalIPs:
  - 10.10.0.82
</code></pre>
<p>logstash_config.yaml</p>
<pre><code>kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: kube-system
data:
  logstash.yml: |
    http.host: &quot;0.0.0.0&quot;
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    input {
      beats {
        port =&gt; 5044
      }
    }
    filter {
      grok {
          match =&gt; { &quot;message&quot; =&gt; &quot;%{COMBINEDAPACHELOG}&quot; }
      }
      date {
        match =&gt; [ &quot;timestamp&quot; , &quot;dd/MMM/yyyy:HH:mm:ss Z&quot; ]
      }
      geoip {
          source =&gt; &quot;clientip&quot;
        }
      }
      output {
        elasticsearch {
          hosts =&gt; [&quot;http://10.10.0.82:9200&quot;]
      }
    }
</code></pre>
<p>logstash deployment</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash-deployment
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:6.3.0
        ports:
        - containerPort: 5044
        volumeMounts:
          - name: config-volume
            mountPath: /usr/share/logstash/config
          - name: logstash-pipeline-volume
            mountPath: /usr/share/logstash/pipeline
      volumes:
      - name: config-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.yml
              path: logstash.yml
      - name: logstash-pipeline-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.conf
              path: logstash.conf
</code></pre>
<p>filebeat.yaml</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [&quot;&quot;] # &quot;&quot; indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      prospectors:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    output.logstash:
      hosts: [&quot;http://10.10.0.82:5044&quot;]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - &quot;*&quot;
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat

    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.8.4
        args: [
          &quot;-c&quot;, &quot;/etc/filebeat.yml&quot;,
          &quot;-e&quot;,
        ]
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: prospectors
          mountPath: /usr/share/filebeat/prospectors.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: prospectors
        configMap:
          defaultMode: 0600
          name: filebeat-prospectors
      - name: data
        emptyDir: {}
</code></pre>
<p>kibana.yaml</p>
<pre><code>kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
spec:
  replicas: 3
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
    spec:
      containers:
      - name: kibana-logging
        image: docker.elastic.co/kibana/kibana-oss:6.8.4
        env:
          - name: ELASTICSEARCH_URL
            value: http://10.10.0.82:9200
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/name: &quot;Kibana&quot;
spec:
  type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
    nodePort: 32010
  selector:
    k8s-app: kibana-logging
</code></pre>
<pre><code>kubectl get svc -n kube-system
elasticsearch      ClusterIP   10.43.50.63    10.10.0.82    9200/TCP                 31m
kibana-logging     NodePort    10.43.58.127   10.10.0.82    5601:32010/TCP           4m4s
kube-dns           ClusterIP   10.43.0.10     &lt;none&gt;        53/UDP,53/TCP,9153/TCP   23d
logstash-service   ClusterIP   10.43.130.36   10.10.0.82    5044/TCP                 30m
</code></pre>
<p>filebeat pod logs :</p>
<pre><code>2020-11-04T16:42:22.857Z        INFO    log/harvester.go:255    Harvester started for file: /var/lib/docker/containers/011d24d334bba573ffbb466b0f3f70ae5ddc986f233e683076eaae7394801203/011d24d334bba573ffbb466b0f3f70ae5ddc986f233e683076eaae7394801203-json.log
2020-11-04T16:42:22.983Z        INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://logstash-service:9600))
2020-11-04T16:42:52.412Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {&quot;monitoring&quot;: {&quot;metrics&quot;: {&quot;beat&quot;:{&quot;cpu&quot;:{&quot;system&quot;:{&quot;ticks&quot;:270,&quot;time&quot;:{&quot;ms&quot;:271}},&quot;total&quot;:{&quot;ticks&quot;:740,&quot;time&quot;:{&quot;ms&quot;:745},&quot;value&quot;:740},&quot;user&quot;:{&quot;ticks&quot;:470,&quot;time&quot;:{&quot;ms&quot;:474}}},&quot;handles&quot;:{&quot;limit&quot;:{&quot;hard&quot;:1048576,&quot;soft&quot;:1048576},&quot;open&quot;:97},&quot;info&quot;:{&quot;ephemeral_id&quot;:&quot;6584086a-eff4-46b5-9be0-93892dad9d97&quot;,&quot;uptime&quot;:{&quot;ms&quot;:30191}},&quot;memstats&quot;:{&quot;gc_next&quot;:36421840,&quot;memory_alloc&quot;:32140904,&quot;memory_total&quot;:55133048,&quot;rss&quot;:65593344}},&quot;filebeat&quot;:{&quot;events&quot;:{&quot;active&quot;:4214,&quot;added&quot;:4219,&quot;done&quot;:5},&quot;harvester&quot;:{&quot;open_files&quot;:89,&quot;running&quot;:88,&quot;started&quot;:88}},&quot;libbeat&quot;:{&quot;config&quot;:{&quot;module&quot;:{&quot;running&quot;:0},&quot;reloads&quot;:2},&quot;output&quot;:{&quot;type&quot;:&quot;logstash&quot;},&quot;pipeline&quot;:{&quot;clients&quot;:2,&quot;events&quot;:{&quot;active&quot;:4117,&quot;filtered&quot;:88,&quot;published&quot;:4116,&quot;total&quot;:4205}}},&quot;registrar&quot;:{&quot;states&quot;:{&quot;current&quot;:5,&quot;update&quot;:5},&quot;writes&quot;:{&quot;success&quot;:6,&quot;total&quot;:6}},&quot;system&quot;:{&quot;cpu&quot;:{&quot;cores&quot;:8},&quot;load&quot;:{&quot;1&quot;:1.9,&quot;15&quot;:0.61,&quot;5&quot;:0.9,&quot;norm&quot;:{&quot;1&quot;:0.2375,&quot;15&quot;:0.0763,&quot;5&quot;:0.1125}}}}}}
2020-11-04T16:42:54.289Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://logstash-service:5044)): dial tcp 10.43.145.162:5044: i/o timeout
2020-11-04T16:42:54.289Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://logstash-service:5044)) with 1 reconnect attempt(s)
</code></pre>
<pre><code>logstash pod logs :
[WARN ] 2020-11-04 15:45:04.648 [Ruby-0-Thread-4: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:232] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=&gt;&quot;http://elasticsearch:9200/&quot;, :error_type=&gt;LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=&gt;&quot;Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch&quot;}
</code></pre>

<p>What I have understood from your architecture, you are using <strong>Filebeat</strong> &gt;&gt; <strong>Logstash</strong> &gt;&gt; <strong>Elasticsearch</strong> &gt;&gt; <strong>Kibana</strong></p>
<p>So, in the <code>filebeat.yml</code>, you have selected output as logstash. But, you have given wrong port for logstash output in <code>filebeat.yml</code>.</p>
<p>It should be:</p>
<pre><code>output.logstash:
  hosts: ['http://195.134.187.25:5044']
</code></pre>
<p>As, if you see in <code>logstash_config.yaml</code>, you have given 5044 as beats input. So, make the changes in <code>filebeat.yml</code> in <code>output.logstash</code></p>
