title: Missing labels in prometheus alerts
body: <p>I'm having issues with Prometheus alerting rules. I have various cAdvisor specific alerts set up, for example:</p>
  <pre><code>- alert: ContainerCpuUsage
    expr: (sum(rate(container_cpu_usage_seconds_total[3m])) BY (instance, name) * 100) &gt; 80
    for: 2m
    labels:
      severity: warning
    annotations:
      title: 'Container CPU usage (instance {{ $labels.instance }})'
      description: 'Container CPU usage is above 80%\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}'
  </code></pre>
  <p>When the condition is met, I can see the alert in the &quot;Alerts&quot; tab in Prometheus, however some labels are missing thus not allowing alertmanager to send a notification via Slack. To be specific, I attach custom &quot;env&quot; label to each target:</p>
  <pre><code> {
    &quot;targets&quot;: [
     &quot;localhost:8080&quot;,
    ],
    &quot;labels&quot;: {
     &quot;job&quot;: &quot;cadvisor&quot;,
     &quot;env&quot;: &quot;production&quot;,
     &quot;__metrics_path__&quot;: &quot;/metrics&quot;
    }
   }
  </code></pre>
  <p>But when the alert based on cadvisor metrics is firing, the labels are: alertname, instance and severity - no job label, no env label.
  All the other alerts from other exporters (f.e. node-exporter) work just fine and the label is present.</p>