<p>I am trying to setup AlertManager for my Kubernetes cluster. I have  followed this document (<a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md" rel="nofollow noreferrer">https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md</a>) -> Everything Ok. </p>

<p>For setting AlertManager, I am studying this document (<a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md" rel="nofollow noreferrer">https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md</a>)</p>

<p>I am getting the <code>CrashLoopBackOff</code> for <code>alertmanager-example-0</code>. Please check the log attached:</p>

<p>1st image : <code>$ kubectl logs -f prometheus-operator-88fcf6d95-zctgw  -n monitoring</code></p>

<p>2nd image : <code>$ kubectl describe pod alertmanager-example-0</code></p>

<p><a href="https://i.stack.imgur.com/YS5rZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YS5rZ.png" alt="enter image description here"></a>
<a href="https://i.stack.imgur.com/8njiB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8njiB.png" alt="enter image description here"></a></p>

<p>Can anyone point out what am I doing wrong? Thanks in advance.</p>

<p>Sounds like you have an issue where <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/" rel="nofollow noreferrer">RBAC</a> and the <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions" rel="nofollow noreferrer">Service Account</a> (<code>system:serviceaccount:monitoring:prometheus-operator</code>) used by your Alert manager pods doesn't have enough permissions to talk to the kube-apiserver. </p>

<p>In your the case of the Prometheus Operator has a ClusterRoleBinding <code>prometheus-operator</code> that looks like this:</p>

<pre><code>$ kubectl get clusterrolebinding prometheus-operator -o=yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: prometheus-operator
  name: prometheus-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-operator
subjects:
- kind: ServiceAccount
  name: prometheus-operator
  namespace: monitoring
</code></pre>

<p>More importantly, the <code>ClusterRole</code> should look something like this:</p>

<pre><code>$ kubectl get clusterrole prometheus-operator -o=yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: prometheus-operator
  name: prometheus-operator
rules:
- apiGroups:
  - extensions
  resources:
  - thirdpartyresources
  verbs:
  - '*'
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - '*'
- apiGroups:
  - monitoring.coreos.com
  resources:
  - alertmanager
  - alertmanagers
  - prometheus
  - prometheuses
  - service-monitor
  - servicemonitors
  - prometheusrules
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - delete
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  verbs:
  - get
  - create
  - update
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - list
  - watch
</code></pre>
