title: Promtail Targets Failed
body: <ul>
  <li>What Grafana version and what operating system are you using?</li>
  </ul>
  <p>Promtail:latest &amp; Loki:2.2.0, Kubernetes (GitVersion:&quot;v1.18.8&quot;) and Helm (Version:&quot;v3.6.2&quot;)</p>
  <ul>
  <li><strong>What</strong> are you trying to achieve?</li>
  </ul>
  <p>To can scrape my active targets and push them to Loki.</p>
  <ul>
  <li>What happened?</li>
  </ul>
  <p>All targets are marked as &quot;not ready&quot;. If I am going to the /targets page, all my active_targets are marked as &quot;false&quot;.
  In Loki I have no logs.
  As well the /var/logs/ folder is empty in Promtail. The logs I am receiving from the promtail pod are like this:</p>
  <pre><code>level=info ts=2021-08-06T05:30:09.076046169Z caller=filetargetmanager.go:254 msg=&quot;Adding target&quot; key=&quot;{app=\&quot;&lt;app_name&gt;\&quot;, container=\&quot;&lt;container_name&gt;&quot;, job=\&quot;&lt;job_name&gt;&quot;, namespace=\&quot;&lt;namesapce_name&gt;&quot;, node_name=\&quot;&lt;nodeName_name&gt;&quot;, pod=\&quot;&lt;pod_name&gt;&quot;}&quot;
  
  level=info ts=2021-08-06T05:30:09.076046169Z caller=filetargetmanager.go:254 msg=&quot;Removing target&quot; key=&quot;{app=\&quot;&lt;app_name&gt;\&quot;, container=\&quot;&lt;container_name&gt;&quot;, job=\&quot;&lt;job_name&gt;&quot;, namespace=\&quot;&lt;namesapce_name&gt;&quot;, node_name=\&quot;&lt;nodeName_name&gt;&quot;, pod=\&quot;&lt;pod_name&gt;&quot;}&quot;
  
  level=info ts=2021-08-06T05:30:14.095615824Z caller=filetarget.go:150 msg=&quot;filetarget: watcher closed, tailer stopped, positions saved&quot; path=/var/log/pods/*&lt;some_path&gt;/&lt;container_name&gt;/*.log
  
  </code></pre>
  <p>Promtail/metrics:</p>
  <pre><code> HELP promtail_targets_failed_total Number of failed targets.
   TYPE promtail_targets_failed_total counter
  promtail_targets_failed_total{reason=&quot;empty_labels&quot;} 2280
  promtail_targets_failed_total{reason=&quot;exists&quot;} 470
   HELP request_duration_seconds Time (in seconds) spent serving HTTP requests.
  </code></pre>
  <ul>
  <li>What did you <strong>expect</strong> to happen?</li>
  </ul>
  <p>That my targets getting scraped and pushed to Loki.</p>
  <ul>
  <li>Can you copy/paste the configuration(s) that you are having problems with?</li>
  </ul>
  <pre><code>
    file: |
      server:
        log_level: {{ .Values.config.logLevel }}
        http_listen_port: {{ .Values.config.serverPort }}
        health_check_target: false
      client:
        url: {{ tpl .Values.config.lokiAddress . }}
        {{- tpl .Values.config.snippets.extraClientConfigs . | nindent 2 }}
      positions:
        filename: /run/promtail/positions.yaml
      scrape_configs:
        {{- tpl .Values.config.snippets.scrapeConfigs . | nindent 2 }}
        {{- tpl .Values.config.snippets.extraScrapeConfigs . | nindent 2 }}
  
  scrapeConfigs: |
        # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
        # Pods with a label 'app.kubernetes.io/name'
        - job_name: kubernetes-pods-app-kubernetes-io-name
          pipeline_stages:
            {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
              target_label: app
            - action: drop
              regex: ''
              source_labels:
                - app
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_component
              target_label: component
            {{- if .Values.config.snippets.addScrapeJobLabel }}
            - action: replace
              replacement: kubernetes-pods-app-kubernetes-io-name
              target_label: scrape_job
            {{- end }}
            {{- toYaml .Values.config.snippets.common | nindent 4 }}
        # Pods with a label 'app'
        - job_name: kubernetes-pods-app
          pipeline_stages:
            {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Drop pods with label 'app.kubernetes.io/name'. They are already considered above
            - action: drop
              regex: .+
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_label_app
              target_label: app
            - action: drop
              regex: ''
              source_labels:
                - app
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_label_component
              target_label: component
            {{- if .Values.config.snippets.addScrapeJobLabel }}
            - action: replace
              replacement: kubernetes-pods-app
              target_label: scrape_job
            {{- end }}
            {{- toYaml .Values.config.snippets.common | nindent 4 }}
        # Pods with direct controllers, such as StatefulSet
        - job_name: kubernetes-pods-direct-controllers
          pipeline_stages:
            {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Drop pods with label 'app.kubernetes.io/name' or 'app'. They are already considered above
            - action: drop
              regex: .+
              separator: ''
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
                - __meta_kubernetes_pod_label_app
            - action: drop
              regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
              source_labels:
                - __meta_kubernetes_pod_controller_name
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_controller_name
              target_label: app
            {{- if .Values.config.snippets.addScrapeJobLabel }}
            - action: replace
              replacement: kubernetes-pods-direct-controllers
              target_label: scrape_job
            {{- end }}
            {{- toYaml .Values.config.snippets.common | nindent 4 }}
        # Pods with indirect controllers, such as Deployment
        - job_name: kubernetes-pods-indirect-controller
          pipeline_stages:
            {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Drop pods with label 'app.kubernetes.io/name' or 'app'. They are already considered above
            - action: drop
              regex: .+
              separator: ''
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
                - __meta_kubernetes_pod_label_app
            - action: keep
              regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
              source_labels:
                - __meta_kubernetes_pod_controller_name
            - action: replace
              regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
              source_labels:
                - __meta_kubernetes_pod_controller_name
              target_label: app
            {{- if .Values.config.snippets.addScrapeJobLabel }}
            - action: replace
              replacement: kubernetes-pods-indirect-controller
              target_label: scrape_job
            {{- end }}
            {{- toYaml .Values.config.snippets.common | nindent 4 }}
        # All remaining pods not yet covered
        - job_name: kubernetes-other
          pipeline_stages:
            {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Drop what has already been covered
            - action: drop
              regex: .+
              separator: ''
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
                - __meta_kubernetes_pod_label_app
            - action: drop
              regex: .+
              source_labels:
                - __meta_kubernetes_pod_controller_name
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_name
              target_label: app
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_label_component
              target_label: component
            {{- if .Values.config.snippets.addScrapeJobLabel }}
            - action: replace
              replacement: kubernetes-other
              target_label: scrape_job
            {{- end }}
            {{- toYaml .Values.config.snippets.common | nindent 4 }}
  
  </code></pre>
  <ul>
  <li><p>Did you receive any errors in the Grafana UI or in related logs? If so, please tell us <strong>exactly</strong> what they were.</p>
  </li>
  <li><p>Did you follow any online instructions? If so, what is the URL?</p>
  </li>
  </ul>
  <p>I followed mostly the instructions of the offical repo.</p>
  <p><a href="https://github.com/grafana/helm-charts/tree/main/charts" rel="nofollow noreferrer">https://github.com/grafana/helm-charts/tree/main/charts</a></p>
  <p>I have created the following recourses:</p>
  <p>For Loki:
  I have a Secret (with the configs), Service and Statefulset.</p>
  <p>Promtail:
  I have a DaemonSet, Secret, powerful ClusterRole and CluserRoleBinding.</p>