<p>I got a kubernetes cluster running on AWS using <a href="https://github.com/kubernetes/kops" rel="nofollow noreferrer">kops</a>. I also got prometheus and grafana set up using <a href="https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus" rel="nofollow noreferrer">kube-prometheus</a>.</p>

<p>What I'm trying to do is to store metrics gathered by prometheus on EBS. My persistent volume claim yaml is:</p>

<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: monitoring

spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
</code></pre>

<p>And prometheus.yaml is:</p>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    beta.kubernetes.io/os: linux
  replicas: 2
  resources:
    requests:
      memory: 400Mi
    volumeMounts:
     - name: prometheus-data
       mountPath: "/data"
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  volumes:
  - name: prometheus-data
    persistentVolumeClaim:
     claimName: prometheus-data
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.4.3
</code></pre>

<p>The 10Gi EBS volume is getting created but it's state remains available. I also tried deleting prometheus pods hoping the data will be retained. Unfortunately that was not the case.</p>

<p>I am able to setup the kube-prometheus with the persistent storage. Please check the following json files:</p>

<p>Promethues-deploy.json</p>

<pre><code>{
"apiVersion": "monitoring.coreos.com/v1",
"kind": "Prometheus",
"metadata": {
    "labels": {
        "prometheus": "k8s"
    },
    "name": "k8s-prom",
    "namespace": "monitoring"
},
"spec": {
    "alerting": {
        "alertmanagers": [
            {
                "name": "alertmanager-main",
                "namespace": "monitoring",
                "port": "web"
            }
        ]
    },
    "baseImage": "quay.io/prometheus/prometheus",
    "replicas": 1,
    "resources": {
        "requests": {
            "memory": "400Mi"
        }
    },
    "ruleSelector": {
        "matchLabels": {
            "prometheus": "k8s",
            "role": "alert-rules"
        }
    },
    "securityContext": {
        "fsGroup": 0,
        "runAsNonRoot": false,
        "runAsUser": 0
    },
    "serviceAccountName": "prometheus",
    "serviceMonitorSelector": {
        "matchExpressions": [
            {
                "key": "k8s-app",
                "operator": "Exists"
            }
        ]
    },
    "storage": {
        "class": "",
        "resources": {},
        "selector": {},
        "volumeClaimTemplate": {
            "spec": {
                "resources": {
                    "requests": {
                        "storage": "10Gi"
                    }
                },
                "selector": {
                    "matchLabels": {
                        "app": "k8s-vol"
                    }
                },
                "storageClassName": "no-provision"
            }
        }
    },
    "version": "v2.2.1"
} }
</code></pre>

<p>Prometheus-pv.json</p>

<pre><code>{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
    "labels": {
        "app": "k8s-vol"
    },
    "name": "prometheus-vol",
    "namespace": "monitoring"
},
"spec": {
    "storageClassName": "no-provision",
    "accessModes": [
        "ReadWriteOnce"
    ],
    "capacity": {
        "storage": "10Gi"
    },
    "hostPath": {
        "path": "/data"
    },
    "persistentVolumeReclaimPolicy": "Retain"
},
"status": {
    "phase": "Bound"
} }
</code></pre>

<p>Hope it helps.</p>
