<blockquote>
<p>prometheus-prometheus-kube-prometheus-prometheus-0   0/2    Terminating   0     4s
alertmanager-prometheus-kube-prometheus-alertmanager-0    0/2     Terminating   0          10s</p>
</blockquote>
<p>After updating EKS cluster to 1.16 from 1.15 everything works fine except these two pods, they keep on terminating and unable to initialise. Hence, prometheus monitoring does not work. I am getting below errors while describing the pods.</p>
<pre><code>Error: failed to start container &quot;prometheus&quot;: Error response from daemon: OCI runtime create failed: container_linux.go:362: creating new parent process caused: container_linux.go:1941: running lstat on namespace path &quot;/proc/29271/ns/ipc&quot; caused: lstat /proc/29271/ns/ipc: no such file or directory: unknown
Error: failed to start container &quot;config-reloader&quot;: Error response from daemon: cannot join network of a non running container: 7e139521980afd13dad0162d6859352b0b2c855773d6d4062ee3e2f7f822a0b3
Error: cannot find volume &quot;config&quot; to mount into container &quot;config-reloader&quot;
Error: cannot find volume &quot;config&quot; to mount into container &quot;prometheus&quot;
</code></pre>
<p>here is my yaml file for the deployment:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  creationTimestamp: &quot;2021-04-30T16:39:14Z&quot;
  deletionGracePeriodSeconds: 600
  deletionTimestamp: &quot;2021-04-30T16:49:14Z&quot;
  generateName: prometheus-prometheus-kube-prometheus-prometheus-
  labels:
    app: prometheus
    app.kubernetes.io/instance: prometheus-kube-prometheus-prometheus
    app.kubernetes.io/managed-by: prometheus-operator
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/version: 2.26.0
    controller-revision-hash: prometheus-prometheus-kube-prometheus-prometheus-56d9fcf57
    operator.prometheus.io/name: prometheus-kube-prometheus-prometheus
    operator.prometheus.io/shard: &quot;0&quot;
    prometheus: prometheus-kube-prometheus-prometheus
    statefulset.kubernetes.io/pod-name: prometheus-prometheus-kube-prometheus-prometheus-0
  name: prometheus-prometheus-kube-prometheus-prometheus-0
  namespace: mo
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: prometheus-prometheus-kube-prometheus-prometheus
    uid: 326a09f2-319c-449d-904a-1dd0019c6d80
  resourceVersion: &quot;9337443&quot;
  selfLink: /api/v1/namespaces/monitoring/pods/prometheus-prometheus-kube-prometheus-prometheus-0
  uid: e2be062f-749d-488e-a6cc-42ef1396851b
spec:
  containers:
  - args:
    - --web.console.templates=/etc/prometheus/consoles
    - --web.console.libraries=/etc/prometheus/console_libraries
    - --config.file=/etc/prometheus/config_out/prometheus.env.yaml
    - --storage.tsdb.path=/prometheus
    - --storage.tsdb.retention.time=10d
    - --web.enable-lifecycle
    - --storage.tsdb.no-lockfile
    - --web.external-url=http://prometheus-kube-prometheus-prometheus.monitoring:9090
    - --web.route-prefix=/
    image: quay.io/prometheus/prometheus:v2.26.0
    imagePullPolicy: IfNotPresent
    name: prometheus
    ports:
    - containerPort: 9090
      name: web
      protocol: TCP
    readinessProbe:
      failureThreshold: 120
      httpGet:
        path: /-/ready
        port: web
        scheme: HTTP
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 3
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /etc/prometheus/config_out
      name: config-out
      readOnly: true
    - mountPath: /etc/prometheus/certs
      name: tls-assets
      readOnly: true
    - mountPath: /prometheus
      name: prometheus-prometheus-kube-prometheus-prometheus-db
    - mountPath: /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
      name: prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: prometheus-kube-prometheus-prometheus-token-mh66q
      readOnly: true
  - args:
    - --listen-address=:8080
    - --reload-url=http://localhost:9090/-/reload
    - --config-file=/etc/prometheus/config/prometheus.yaml.gz
    - --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    - --watched-dir=/etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
    command:
    - /bin/prometheus-config-reloader
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: SHARD
      value: &quot;0&quot;
    image: quay.io/prometheus-operator/prometheus-config-reloader:v0.47.0
    imagePullPolicy: IfNotPresent
    name: config-reloader
    ports:
    - containerPort: 8080
      name: reloader-web
      protocol: TCP
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 100m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /etc/prometheus/config
      name: config
    - mountPath: /etc/prometheus/config_out
      name: config-out
    - mountPath: /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
      name: prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: prometheus-kube-prometheus-prometheus-token-mh66q
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: prometheus-prometheus-kube-prometheus-prometheus-0
  nodeName: ip-10-1-49-45.ec2.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccount: prometheus-kube-prometheus-prometheus
  serviceAccountName: prometheus-kube-prometheus-prometheus
  subdomain: prometheus-operated
  terminationGracePeriodSeconds: 600
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: config
    secret:
      defaultMode: 420
      secretName: prometheus-prometheus-kube-prometheus-prometheus
  - name: tls-assets
    secret:
      defaultMode: 420
      secretName: prometheus-prometheus-kube-prometheus-prometheus-tls-assets
  - emptyDir: {}
    name: config-out
  - configMap:
      defaultMode: 420
      name: prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
    name: prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
  - emptyDir: {}
    name: prometheus-prometheus-kube-prometheus-prometheus-db
  - name: prometheus-kube-prometheus-prometheus-token-mh66q
    secret:
      defaultMode: 420
      secretName: prometheus-kube-prometheus-prometheus-token-mh66q
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: &quot;2021-04-30T16:39:14Z&quot;
    status: &quot;True&quot;
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
</code></pre>

<p>If someone needs to know the answer, in my case(the above situation) there were 2 Prometheus operators running in different different namespace, 1 in default &amp; another monitoring namespace. so I removed the one from the default namespace and it resolved my pods crashing issue.</p>
