<p>We have elasticsearch cluster at <code>${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}</code>
and filebeat pod at k8s cluster that exports other pods' logs</p>
<p>There is <code>filebeat.yml</code>:</p>
<pre><code>filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.namespace: develop
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log
              exclude_lines: [&quot;^\\s+[\\-`('.|_]&quot;]
      hints.enabled: true
      hints.default_config:
        type: container
        multiline.type: pattern
        multiline.pattern: '^[[:space:]]'
        multiline.negate: false
        multiline.match: after
http:
  enabled: true
  host: localhost
  port: 5066

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
  indices:
    - index: &quot;develop&quot;
      when:
        equals:
          kubernetes.namespace: &quot;develop&quot;
    - index: &quot;kubernetes-dev&quot;
      when:
        not:
          and:
          - equals:
              kubernetes.namespace: &quot;develop&quot;

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
  - add_kubernetes_metadata:
    host: ${NODE_NAME}
    matchers:
      - logs_path:
          logs_path: &quot;/var/log/containers/&quot;
  - decode_json_fields:
      fields: [&quot;message&quot;]
      add_error_key: true
      process_array: true
      overwrite_keys: false
      max_depth: 10
      target: json_message
</code></pre>
<p>I've checked: filebeat has access to <code>/var/log/containers/</code> on kuber but elastic cluster still doesn't get any  <code>develop</code> or <code>kubernetes-dev</code> indices. (Cluster has relative index templates for this indices)</p>
<p><code>http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cluster/health?pretty</code>:</p>
<pre><code>{
  &quot;cluster_name&quot; : &quot;elasticsearch&quot;,
  &quot;status&quot; : &quot;green&quot;,
  &quot;timed_out&quot; : false,
  &quot;number_of_nodes&quot; : 3,
  &quot;number_of_data_nodes&quot; : 3,
  &quot;active_primary_shards&quot; : 14,
  &quot;active_shards&quot; : 28,
  &quot;relocating_shards&quot; : 0,
  &quot;initializing_shards&quot; : 0,
  &quot;unassigned_shards&quot; : 0,
  &quot;delayed_unassigned_shards&quot; : 0,
  &quot;number_of_pending_tasks&quot; : 0,
  &quot;number_of_in_flight_fetch&quot; : 0,
  &quot;task_max_waiting_in_queue_millis&quot; : 0,
  &quot;active_shards_percent_as_number&quot; : 100.0
}
</code></pre>
<p>Filebeat log:</p>
<pre><code>{
    &quot;log.level&quot;: &quot;info&quot;,
    &quot;@timestamp&quot;: &quot;2022-11-25T08:35:18.084Z&quot;,
    &quot;log.logger&quot;: &quot;monitoring&quot;,
    &quot;log.origin&quot;: {
        &quot;file.name&quot;: &quot;log/log.go&quot;,
        &quot;file.line&quot;: 184
    },
    &quot;message&quot;: &quot;Non-zero metrics in the last 30s&quot;,
    &quot;service.name&quot;: &quot;filebeat&quot;,
    &quot;monitoring&quot;: {
        &quot;metrics&quot;: {
            &quot;beat&quot;: {
                &quot;cgroup&quot;: {
                    &quot;cpu&quot;: {
                        &quot;stats&quot;: {
                            &quot;periods&quot;: 38
                        }
                    },
                    &quot;cpuacct&quot;: {
                        &quot;total&quot;: {
                            &quot;ns&quot;: 1576170001
                        }
                    },
                    &quot;memory&quot;: {
                        &quot;mem&quot;: {
                            &quot;usage&quot;: {
                                &quot;bytes&quot;: 4096
                            }
                        }
                    }
                },
                &quot;cpu&quot;: {
                    &quot;system&quot;: {
                        &quot;ticks&quot;: 13570,
                        &quot;time&quot;: {
                            &quot;ms&quot;: 56
                        }
                    },
                    &quot;total&quot;: {
                        &quot;ticks&quot;: 23320,
                        &quot;time&quot;: {
                            &quot;ms&quot;: 90
                        },
                        &quot;value&quot;: 23320
                    },
                    &quot;user&quot;: {
                        &quot;ticks&quot;: 9750,
                        &quot;time&quot;: {
                            &quot;ms&quot;: 34
                        }
                    }
                },
                &quot;handles&quot;: {
                    &quot;limit&quot;: {
                        &quot;hard&quot;: 1048576,
                        &quot;soft&quot;: 1048576
                    },
                    &quot;open&quot;: 11
                },
                &quot;info&quot;: {
                    &quot;ephemeral_id&quot;: &quot;a88f461e-1fd3-48d3-a1c7-3bfea1124ab9&quot;,
                    &quot;uptime&quot;: {
                        &quot;ms&quot;: 7530198
                    },
                    &quot;version&quot;: &quot;8.1.0&quot;
                },
                &quot;memstats&quot;: {
                    &quot;gc_next&quot;: 17767824,
                    &quot;memory_alloc&quot;: 16385608,
                    &quot;memory_total&quot;: 709250600,
                    &quot;rss&quot;: 104206336
                },
                &quot;runtime&quot;: {
                    &quot;goroutines&quot;: 23
                }
            },
            &quot;filebeat&quot;: {
                &quot;harvester&quot;: {
                    &quot;open_files&quot;: 0,
                    &quot;running&quot;: 0
                }
            },
            &quot;libbeat&quot;: {
                &quot;config&quot;: {
                    &quot;module&quot;: {
                        &quot;running&quot;: 0
                    }
                },
                &quot;output&quot;: {
                    &quot;events&quot;: {
                        &quot;active&quot;: 0
                    }
                },
                &quot;pipeline&quot;: {
                    &quot;clients&quot;: 0,
                    &quot;events&quot;: {
                        &quot;active&quot;: 0
                    }
                }
            },
            &quot;registrar&quot;: {
                &quot;states&quot;: {
                    &quot;current&quot;: 0
                }
            },
            &quot;system&quot;: {
                &quot;load&quot;: {
                    &quot;1&quot;: 3.99,
                    &quot;15&quot;: 3.91,
                    &quot;5&quot;: 4.08,
                    &quot;norm&quot;: {
                        &quot;1&quot;: 0.4988,
                        &quot;15&quot;: 0.4888,
                        &quot;5&quot;: 0.51
                    }
                }
            }
        },
        &quot;ecs.version&quot;: &quot;1.6.0&quot;
    }
}
</code></pre>
<p>Where can be problem? Im absolutely new to elk so working with it not from the start is a bit hard</p>

<p>In the end I've found out that elastic cluster's version used by company is much older than that of filebeat (8.1 against 7.9). So temporary allowing usage of older versions has solved the issue for time being.</p>
<pre><code>output.elasticsearch:
    allow_older_versions: true
</code></pre>
<p>BUT @Paulo also had a point and construction <code>not + and</code> was excessive in the end.</p>
