<p>I am currently attempting to get the logs from my kubernetes cluster to an external ElasticSearch/Kibana. So far I have used <a href="https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml" rel="nofollow noreferrer">this</a> daemon deployment to get filebeat running and piping to my external server, but I am unable to figure out how to set the index to something meaningfull. <a href="https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html" rel="nofollow noreferrer">This</a> documentation page tells me that I need to create am index key in the output.elasticsearch section, but I don't know what to put in the value. </p>

<p>My desired output format would be something along the lines of <code>&lt;cluster-name&gt;-&lt;namespace&gt;-&lt;pod name&gt;</code></p>

<p>IE: <code>devKube-frontend-publicAPI-123abc</code></p>

deef0000dragon1
<p>Precondition: You have enabled <a href="https://www.elastic.co/guide/en/beats/filebeat/current/add-kubernetes-metadata.html" rel="nofollow noreferrer"><code>add_kubernetes_metadata: ~</code></a>.</p>

<p>Then you can use that metadata in the index name like this:</p>

<pre><code>output.elasticsearch:
  index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
</code></pre>

<ul>
<li><code>%{[kubernetes.namespace]:filebeat}</code>: Use the Kubernetes namespace or if there is none fall back to <code>filebeat</code>.</li>
<li>Adding <code>%{[beat.version]}</code> is highly recommended for the scenario when you upgrade Filebeat and there is a breaking change in the mapping. This should be limited to major version changes (if at all), but is an issue you can easily avoid with this setting.</li>
<li>A time based index pattern like <code>%{+yyyy.MM.dd}</code> or even better would be an <a href="https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html" rel="nofollow noreferrer">ILM policy</a> to have evenly and properly sized shards.</li>
</ul>

<p>PS: You have the Pod name and other details in fields from <code>add_kubernetes_metadata: ~</code>. I would be careful not to cut the indices into tiny pieces, since every shard has a certain amount of overhead. The default Filebeat ILM policy is 50GB per shard — if your shards are smaller than 10GB you will most likely run into issues at some point. Leave the indices a bit more coarse grained and just use a filter for a specific Pod instead.</p>
