<p>I have set up Grafana, Prometheus and loki (2.6.1) as follows on my kubernetes (1.21) cluster:</p>
<pre class="lang-bash prettyprint-override"><code>helm upgrade --install promtail grafana/promtail -n monitoring -f monitoring/promtail.yaml
helm upgrade --install prom prometheus-community/kube-prometheus-stack -n monitoring --values monitoring/prom.yaml
helm upgrade --install loki grafana/loki -n monitoring --values monitoring/loki.yaml
</code></pre>
<p>with:</p>
<pre class="lang-yaml prettyprint-override"><code># monitoring/loki.yaml
loki:
  schemaConfig:
    configs:
    - from: 2020-09-07
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

  storageConfig:
    aws:
      s3: s3://eu-west-3/cluster-loki-logs
    boltdb_shipper:
      shared_store: filesystem
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 168h

# monitoring/promtail.yaml
config:
  serverPort: 80
  clients:
    - url: http://loki:3100/loki/api/v1/push

# monitoring/prom.yaml
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector:
      matchLabels:
        monitored: &quot;true&quot;

grafana:
  sidecar:
    datasources:
      defaultDatasourceEnabled: true
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki.monitoring:3100
</code></pre>
<p>I get data from my containers, but, whenever I have a container logging in json format, I can't get access to the nested fields:</p>
<pre><code>{app=&quot;product&quot;, namespace=&quot;api-dev&quot;} | unpack | json
</code></pre>
<p>Yields:</p>
<p><a href="https://i.stack.imgur.com/LIN5h.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LIN5h.png" alt="enter image description here" /></a></p>
<p>My aim is, for example, to filter by <code>log.severity</code></p>

GlinesMome
<p>Actually, following <a href="https://stackoverflow.com/a/72661706/1599054">this answer</a>, it occurs to be a <code>promtail</code> scraping issue.</p>
<p>The current (<code>promtail-6.3.1</code> / 2.6.1) helm chart default is to have <a href="https://grafana.com/docs/loki/latest/clients/promtail/stages/cri/" rel="nofollow noreferrer"><code>cri</code></a> as pipeline's stage, which expects this kind of logs:</p>
<pre><code>&quot;2019-04-30T02:12:41.8443515Z stdout xx message&quot;
</code></pre>
<p>I should have use <a href="https://grafana.com/docs/loki/latest/clients/promtail/stages/docker/" rel="nofollow noreferrer"><code>docker</code></a>, which expects json, consequently, my <code>promtail.yaml</code> changed to:</p>
<pre><code>config:
  serverPort: 80
  clients:
    - url: http://loki:3100/loki/api/v1/push
  snippets:
    pipelineStages:
      - docker: {}
</code></pre>
