<p>I'm trying to write logs to an Elasticsearch index from a Kubernetes cluster.  Fluent-bit is being used to read stdout and it enriches the logs with metadata including pod labels.  A simplified example log object is</p>

<pre><code>{
  "log": "This is a log message.",
  "kubernetes": {
    "labels": {
      "app": "application-1"
    }
  }
}
</code></pre>

<p>The problem is that a few other applications deployed to the cluster have labels of the following format:</p>

<pre><code>{
  "log": "This is another log message.",
  "kubernetes": {
    "labels": {
      "app.kubernetes.io/name": "application-2"
    }
  }
}
</code></pre>

<p>These applications are installed via Helm charts and the newer ones are following the label and selector conventions as laid out <a href="https://github.com/helm/charts/blob/master/REVIEW_GUIDELINES.md#metadata" rel="noreferrer">here</a>.  The naming convention for labels and selectors was updated in Dec 2018, seen <a href="https://github.com/helm/charts/commit/7458584650756b9ba8e9380ac727e89fe5ae285c#diff-9b00c37f18653dae7f2730865280ae4c" rel="noreferrer">here</a>, and not all charts have been updated to reflect this.</p>

<p>The end result of this is that depending on which type of label format makes it into an Elastic index first, trying to send the other type in will throw a mapping exception.  If I create a new empty index and send in the namespaced label first, attempting to log the simple <code>app</code> label will throw this exception:</p>

<pre><code>object mapping for [kubernetes.labels.app] tried to parse field [kubernetes.labels.app] as object, but found a concrete value
</code></pre>

<p>The opposite situation, posting the namespaced label second, results in this exception:</p>

<pre><code>Could not dynamically add mapping for field [kubernetes.labels.app.kubernetes.io/name]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].
</code></pre>

<p>What I suspect is happening is that Elasticsearch sees the periods in the field name as JSON dot notation and is trying to flesh it out as an object.  I was able to find <a href="https://github.com/elastic/elasticsearch/commit/aed1f68e494c65ad50b98a3e0a7a2b6a794b2965" rel="noreferrer">this PR</a> from 2015 which explicitly disallows periods in field names however it seems to have been reversed in 2016 with <a href="https://github.com/elastic/elasticsearch/pull/17759" rel="noreferrer">this PR</a>.  There is also this multi-year <a href="https://discuss.elastic.co/t/field-name-cannot-contain/33251/48" rel="noreferrer">thread</a> from 2015-2017 discussing this issue but I was unable to find anything recent involving the latest versions.</p>

<p>My current thoughts on moving forward is to standardize the Helm charts we are using to have all of the labels use the same convention.  This seems like a band-aid on the underlying issue though which is that I feel like I'm missing something obvious in the configuration of Elasticsearch and dynamic field mappings.</p>

<p>Any help here would be appreciated.</p>

<p>I opted to use the Logstash mutate filter with the <code>rename</code> option as described here:</p>

<p><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename" rel="nofollow noreferrer">https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename</a></p>

<p>The end result looked something like this:</p>

<pre><code>filter {
  mutate {
    '[kubernetes][labels][app]'   =&gt; '[kubernetes][labels][app.kubernetes.io/name]'
    '[kubernetes][labels][chart]' =&gt; '[kubernetes][labels][helm.sh/chart]'
  }
}
</code></pre>
