<p>We are trying to monitor K8S with Grafana and Prometheus Operator. Most of the metrics are working as expected and I was able to see the dashboard with the right value, our system contain 10 nodes with overall 500 pods. Now when I restarted Prometheus all the data was <strong>deleted</strong>. I want it to be stored for two week.</p>
<p>My question is, How can I define to Prometheus volume to keep the data for two weeks or 100GB DB.</p>
<p>I found the following (we use <a href="https://github.com/coreos/prometheus-operator" rel="nofollow noreferrer">Prometheus</a> operator):</p>
<p><a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/storage.md" rel="nofollow noreferrer">https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/storage.md</a></p>
<h3>This is the config of the Prometheus Operator</h3>
<pre><code>apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    k8s-app: prometheus-operator
  name: prometheus-operator
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: prometheus-operator
  template:
    metadata:
      labels:
        k8s-app: prometheus-operator
    spec:
      containers:
      - args:
        - --kubelet-service=kube-system/kubelet
        - --logtostderr=true
        - --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
        - --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.29.0
        image: quay.io/coreos/prometheus-operator:v0.29.0
        name: prometheus-operator
        ports:
        - containerPort: 8080
          name: http
</code></pre>
<h3>This is the config of the Prometheus</h3>
<pre><code>    apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
      name: prometheus
      namespace: monitoring
      labels: 
        prometheus: prometheus
    spec:
      replica: 2
      serviceAccountName: prometheus
      serviceMonitorNamespaceSelector: {}
      serviceMonitorSelector:
        matchLabels:
          role: observeable
      tolerations:
      - key: &quot;WorkGroup&quot;
        operator: &quot;Equal&quot;
        value: &quot;operator&quot;
        effect: &quot;NoSchedule&quot;
      - key: &quot;WorkGroup&quot;
        operator: &quot;Equal&quot;
        value: &quot;operator&quot;
        effect: &quot;NoExecute&quot;
      resources:
        limits:
          cpu: 8000m
          memory: 24000Mi
        requests:
          cpu: 6000m
          memory: 6000Mi
     storage:
       volumeClaimTemplate:
         spec:
        selector:
          matchLabels:
            app: prometheus
        resources:
          requests:
            storage: 100Gi

</code></pre>
<p><a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/storage.md" rel="nofollow noreferrer">https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/storage.md</a></p>
<p>We have file system (nfs), and the above storage config doesn't works,
my questions are:</p>
<ol>
<li>What I miss here is how to config the <code>volume</code>, <code>server</code> , <code>path</code> in the following its under the <code>nfs</code> section? Where should I find this <code>/path/to/prom/db</code>? How can I refer to it? Should I create it somehow, or just provide the path?</li>
</ol>
<p>We have NFS configured in our system.</p>
<ol start="2">
<li>How to combine it to Prometheus?</li>
</ol>
<p>As I don't have deep knowledge in <code>pvc</code> and <code>pv</code>, I've created the following (not sure regard those values, what is my server and what path should I provide)...</p>
<pre class="lang-yaml prettyprint-override"><code>server: myServer
path: &quot;/path/to/prom/db&quot;
</code></pre>
<p>What should I put there and how I make <strong>my Prometheus</strong> (i.e. the config I have provided in the question) to use it?</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
    prometheus: prometheus
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce # required
  nfs:
    server: myServer
    path: &quot;/path/to/prom/db&quot;
</code></pre>
<p>If there any other persistence volume other than <code>nfs</code> which I can use for my use-case? Please advice how.</p>

JME
<p>I started working with the operator chart recently ,</p>

<p>And managed to add persistency without defining pv and pvc.</p>

<p>On the new chart configuration adding persistency is much easier than you describe just edit the file  /helm/vector-chart/prometheus-operator-chart/<strong>values.yaml</strong>  under prometheus.prometheusSpec:</p>

<pre><code>storageSpec:
  volumeClaimTemplate:
    spec:
      storageClassName: prometheus
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
    selector: {}
</code></pre>

<p>And add this /helm/vector-chart/prometheus-operator-chart/templates/prometheus/<strong>storageClass.yaml</strong>:</p>

<pre><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: prometheus
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
parameters:
  type: gp2
  zones: "ap-southeast-2a, ap-southeast-2b, ap-southeast-2c"
  encrypted: "true"
</code></pre>

<p>This will automatically create you both <strong>pv</strong> and a <strong>pvc</strong> which will create an ebs in aws which will store all your data inside.</p>
