<p>I am trying to collect logs from the running pod in the KOPS cluster. I run filebeat DemonSet in the KOPS cluster to collect logs from my pod(application) and then ship those logs to the outside of the cluster where the logstash service is accepting them and saves them into a file.</p>
<p>I noticed filebeat always producing the logs with UTC timestamp even though all of my nodes and pods are running in SGT timezone.</p>
<p>I set add_locale in filebeat processor but it doesn't help.</p>
<ul>
<li>add_locale:
format: offset</li>
</ul>
<p>nodes timezone</p>
<p><a href="https://i.stack.imgur.com/QbyDb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QbyDb.png" alt="enter image description here" /></a></p>
<p>pod timezone</p>
<p><a href="https://i.stack.imgur.com/ii3SB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ii3SB.png" alt="enter image description here" /></a></p>
<p>Complete filebeat-kubernetes.yaml</p>
<pre><code>---
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: &quot;application1&quot;
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}*.log
            - condition:
                contains:
                  kubernetes.pod.name: &quot;application2&quot;
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}*.log
    processors:
      - add_locale:
          format: offset
      - add_kubernetes_metadata:
          host: ${NODE_NAME} 
          matchers:
            - logs_path:
                logs_path: &quot;/var/log/containers/&quot;
    output.logstash:
      hosts: [&quot;IP:5044&quot;]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.10.1
        args: [
          &quot;-c&quot;, &quot;/etc/filebeat.yml&quot;,
          &quot;-e&quot;,
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: tz-config
          mountPath: /etc/localtime
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
      - name: tz-config
        hostPath:
          path: /usr/share/zoneinfo/Singapore

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [&quot;&quot;] # &quot;&quot; indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
---
</code></pre>
<p>output log from filebeat</p>
<p><a href="https://i.stack.imgur.com/m569P.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m569P.png" alt="enter image description here" /></a></p>

Kyaw Min Thu L
<p>Unfortunately, I don't have enough reputation to add a comment so posting this as an answer. The <a href="https://www.elastic.co/guide/en/beats/filebeat/current/add-locale.html" rel="nofollow noreferrer">following reference doc</a> mentions:</p>
<blockquote>
<p>The processor adds the a event.timezone value to each event.</p>
</blockquote>
<p>so it could be possible that the log timestamp itself is not converted to the local timezone but it adds additional field in the event logs to represent the timezone and that can be used to format the logs by the application consuming the logs.</p>
