<p>Disclaimer: First time I use Prometheus.</p>

<p>I am trying to send a Slack notification every time a Job ends successfully. </p>

<p>To achieve this, I installed kube-state-metrics, Prometheus and AlertManager.</p>

<p>Then I created the following rule:</p>

<pre><code>rules:
  - alert: KubeJobCompleted
    annotations:    
      identifier: '{{ $labels.instance }}'
      summary: Job Completed Successfully
      description: Job *{{ $labels.namespace }}/{{ $labels.job_name }}* is completed successfully.
    expr: |
      kube_job_spec_completions{job="kube-state-metrics"} - kube_job_status_succeeded{job="kube-state-metrics"}  == 0
    labels:
      severity: information
</code></pre>

<p>And added the AlertManager receiver text (template) :</p>

<pre><code>{{ define "custom_slack_message" }}
{{ range .Alerts }}
    {{ .Annotations.description }}
{{ end }} 
{{ end }} 
</code></pre>

<p>My current result: Everytime a new job completes successfully, I receive a Slack notification with the list of all Job that completed successfully.</p>

<p>I don't mind receiving the whole list at first but after that I would like to receive notifications that contain only the newly completed job(s) in the specified group interval.</p>

<p>Is it possible?</p>

ThatChrisGuy
<p>Just add extra rule which will just display last completed job(s):</p>

<p>line: <code>for: &lt;10m&gt;</code> - which will list just lastly completed job(s) in 10 minutes:</p>

<pre><code>rules:
  - alert: KubeJobCompleted
    annotations:    
      identifier: '{{ $labels.instance }}'
      summary: Job Completed Successfully
      description: Job *{{ $labels.namespace }}/{{ $labels.job_name }}* is completed successfully.
    expr: |
      kube_job_spec_completions{job="kube-state-metrics"} - kube_job_status_succeeded{job="kube-state-metrics"}  == 0
      for: 10m
    labels:
      severity: information
</code></pre>
