<p>I created a pvc, which dynamically creates a persistenvolume (using k3s with local-path) that gets used by a deployment. I am provisioning everything using terraform but encountered an error. The terraform apply enters a infinite loop while creating the pvc and pod. The pvc is in this state:</p>
<pre><code>Name:          grafana-pvc
Namespace:     default
StorageClass:  local-path
Status:        Pending
Volume:
Labels:        io.kompose.service=grafana-data
Annotations:   &lt;none&gt;
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       grafana-778c7f77c7-w7x9f
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  WaitForFirstConsumer  79s               persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled   7s (x5 over 67s)  persistentvolume-controller  waiting for pod grafana-778c7f77c7-w7x9f to be scheduled
</code></pre>
<p>and the pod is in this state:</p>
<pre><code>Name:             grafana-778c7f77c7-w7x9f
Namespace:        default
Priority:         0
Service Account:  default
Node:             &lt;none&gt;
Labels:           io.kompose.service=grafana
                  pod-template-hash=778c7f77c7
Annotations:      &lt;none&gt;
Status:           Pending
IP:
IPs:              &lt;none&gt;
Controlled By:    ReplicaSet/grafana-778c7f77c7
Containers:
  grafana:
    Image:        grafana/grafana:9.2.4
    Port:         3000/TCP
    Host Port:    0/TCP
    Environment:  &lt;none&gt;
    Mounts:
      /etc/grafana from grafana-configuration (rw)
      /var/lib/grafana from grafana-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n7cmt (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  grafana-configuration:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana-configuration
    ReadOnly:   false
  grafana-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana-pvc
    ReadOnly:   false
  kube-api-access-n7cmt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              &lt;none&gt;
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  90s   default-scheduler  0/1 nodes are available: 1 persistentvolumeclaim &quot;grafana-configuration&quot; not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  89s   default-scheduler  0/1 nodes are available: 1 persistentvolumeclaim &quot;grafana-configuration&quot; not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
</code></pre>
<p>At this stage, nothing works anymore, terraform times out and I am not able to restore the state anymore.</p>
<p>My terraform files:</p>
<p>grafana.tf:</p>
<pre><code>resource &quot;kubernetes_persistent_volume_claim&quot; &quot;grafana-configuration&quot; {
  metadata {
    name = &quot;grafana-configuration&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;grafana-configuration&quot;
    }
  }

  spec {
    access_modes = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;1Gi&quot;
      }
    }
    volume_name = &quot;grafana-configuration&quot;
  }
}

resource &quot;kubernetes_persistent_volume&quot; &quot;grafana-configuration&quot; {
  metadata {
    name = &quot;grafana-configuration&quot;
  }
  spec {
    storage_class_name = &quot;local-path&quot;
    access_modes = [&quot;ReadWriteOnce&quot;]
    capacity = {
      storage = &quot;1Gi&quot;
    }
    node_affinity {
      required {
        node_selector_term {
          match_expressions {
            key = &quot;node-role.kubernetes.io/master&quot;
            operator = &quot;In&quot;
            values = [&quot;true&quot;]
          }
        }
      } 
    }
    persistent_volume_source {
      local {
        path = &quot;/home/administrator/Metrics.Infrastructure/grafana/&quot;
      }
    }
  }
}

resource &quot;kubernetes_persistent_volume_claim&quot; &quot;grafana-pvc&quot; {
  metadata {
    name = &quot;grafana-pvc&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;grafana-data&quot;
    }
  }

  spec {
    access_modes = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;5Gi&quot;
      }
    }
  }
}

resource &quot;kubernetes_deployment&quot; &quot;grafana&quot; {
  metadata {
    name = &quot;grafana&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;grafana&quot;
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        &quot;io.kompose.service&quot; = &quot;grafana&quot;
      }
    }

    template {
      metadata {
        labels = {
          &quot;io.kompose.service&quot; = &quot;grafana&quot;
        }
      }

      spec {
        volume {
          name = &quot;grafana-configuration&quot;
          persistent_volume_claim {
            claim_name = &quot;grafana-configuration&quot;
          }
        }

        volume {
          name = &quot;grafana-data&quot;

          persistent_volume_claim {
            claim_name = &quot;grafana-pvc&quot;
          }
        }

        container {
          name  = &quot;grafana&quot;
          image = &quot;grafana/grafana:9.2.4&quot;

          port {
            container_port = 3000
          }

          volume_mount {
            name       = &quot;grafana-configuration&quot;
            mount_path = &quot;/etc/grafana&quot;
          }
          volume_mount {
            name       = &quot;grafana-data&quot;
            mount_path = &quot;/var/lib/grafana&quot;
          }
        }

        restart_policy = &quot;Always&quot;
      }
    }

    strategy {
      type = &quot;Recreate&quot;
    }
  }
}

resource &quot;kubernetes_service&quot; &quot;grafana&quot; {
    metadata {
        name = &quot;grafana&quot;

        labels = {
            &quot;io.kompose.service&quot; = &quot;grafana&quot;
        }
    }


    spec {
        port {
            port        = 3000
            target_port = 3000
            node_port = 30001
        }
        type = &quot;NodePort&quot;

        selector = {
            &quot;io.kompose.service&quot; = &quot;grafana&quot;
        }
    }
}

</code></pre>
<p>prometheus.tf:</p>
<pre><code># We need these resources so that prometheus can fetch kubernetes metrics
resource &quot;kubernetes_cluster_role&quot; &quot;prometheus-clusterrole&quot; {
  metadata {
    name = &quot;prometheus-clusterrole&quot;
  }

  rule {
    api_groups = [&quot;&quot;]
    resources  = [&quot;nodes&quot;, &quot;nodes/proxy&quot;, &quot;services&quot;, &quot;endpoints&quot;, &quot;pods&quot;]
    verbs      = [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  }
  rule {
    api_groups = [&quot;extensions&quot;]
    resources  = [&quot;ingresses&quot;]
    verbs      = [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  }
  rule {
    non_resource_urls = [&quot;/metrics&quot;]
    verbs      = [&quot;get&quot;]
  }
}

resource &quot;kubernetes_cluster_role_binding&quot; &quot;prometheus_clusterrolebinding&quot; {
  metadata {
    name = &quot;prometheus-clusterrolebinding&quot;
  }
  role_ref {
    api_group = &quot;rbac.authorization.k8s.io&quot;
    kind      = &quot;ClusterRole&quot;
    name      = &quot;prometheus-clusterrole&quot;
  }
  subject {
    kind      = &quot;ServiceAccount&quot;
    name      = &quot;default&quot;
    namespace = &quot;default&quot;
  }
}



resource &quot;kubernetes_config_map&quot; &quot;prometheus-config&quot; {
    metadata {
      name = &quot;prometheus-config&quot;
    }
    data = {
        &quot;prometheus.yml&quot; = &quot;${file(&quot;${path.module}/prometheus/prometheus.yml&quot;)}&quot;
    }
}

resource &quot;kubernetes_persistent_volume_claim&quot; &quot;prometheus_data_claim&quot; {
  metadata {
    name = &quot;prometheus-data-claim&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;prometheus-data&quot;
    }
  }

  spec {
    access_modes = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;20Gi&quot;
      }
    }
  }
}

resource &quot;kubernetes_deployment&quot; &quot;prometheus&quot; {
  metadata {
    name = &quot;prometheus&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;prometheus&quot;
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        &quot;io.kompose.service&quot; = &quot;prometheus&quot;
      }
    }

    template {
      metadata {
        labels = {
          &quot;io.kompose.service&quot; = &quot;prometheus&quot;
        }
      }

      spec {

        volume {
          name = &quot;prometheus-data&quot;

          persistent_volume_claim {
            claim_name = &quot;prometheus-data-claim&quot;
          }
        }

        volume {
          name = &quot;prometheus-config&quot;
          config_map {
            name = &quot;prometheus-config&quot;
          }
        }

        container {
          name  = &quot;prometheus&quot;
          image = &quot;prom/prometheus:v2.40.0&quot;
          args  = [
            &quot;--config.file=/config/prometheus.yml&quot;, 
            &quot;--storage.tsdb.path=/prometheus&quot;,
            &quot;--web.enable-lifecycle&quot;
          ]

          port {
            container_port = 9090
          }

          volume_mount {
            name       = &quot;prometheus-config&quot;
            mount_path = &quot;/config&quot;
          }

          volume_mount {
            name       = &quot;prometheus-data&quot;
            mount_path = &quot;/prometheus&quot;
          }
        }

        restart_policy = &quot;Always&quot;
      }
    }

    strategy {
      type = &quot;Recreate&quot;
    }
  }
}

resource &quot;kubernetes_service&quot; &quot;prometheus&quot; {
    metadata {
        name = &quot;prometheus&quot;

        labels = {
            &quot;io.kompose.service&quot; = &quot;prometheus&quot;
        }
    }

    spec {
        port {
            port        = 80
            target_port = 9090
            node_port = 30000
        }
        type = &quot;NodePort&quot;

        selector = {
            &quot;io.kompose.service&quot; = &quot;prometheus&quot;
        }
    }
}

</code></pre>

kaffarell
<p>Instead of hardcoding the PVC names, you can use exported attributes to create implicit dependencies in terraform. That way terraform will know in which order to create resources and it will help you avoid needing to type the names all the time. For brevity:</p>
<pre><code>        volume {
          name = &quot;grafana-configuration&quot;
          persistent_volume_claim {
            claim_name = kubernetes_persistent_volume_claim.grafana-configuration.metadata.0.name
          }
        }

        volume {
          name = &quot;grafana-data&quot;

          persistent_volume_claim {
            claim_name = kubernetes_persistent_volume_claim.grafana-pvc.metadata.0.name
          }
        }
</code></pre>
<p>Terraform plan output for reference:</p>
<pre class="lang-bash prettyprint-override"><code>                  + volume {
                      + name = &quot;grafana-configuration&quot;

                      + persistent_volume_claim {
                          + claim_name = &quot;grafana-configuration&quot;
                          + read_only  = false
                        }
                    }
                  + volume {
                      + name = &quot;grafana-data&quot;

                      + persistent_volume_claim {
                          + claim_name = &quot;grafana-configuration&quot;
                          + read_only  = false
                        }
                    }
</code></pre>
<p>Additionally, I would strongly suggest moving to a newer version of the resources which are denoted with <code>_v1</code> at the end of the resource name, e.g., for <a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim_v1" rel="nofollow noreferrer">PVC</a>.</p>
<p>EDIT: As per my comment, in order to make sure the volume is created before the PVC, a similar piece of code can be used to create implicit dependency between the <code>grafana-configuration</code> PV and PVC (shortened for brevity):</p>
<pre><code>  spec {
    access_modes       = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;1Gi&quot;
      }
    }
    volume_name = kubernetes_persistent_volume.grafana-configuration.metadata.0.name # &lt;--- This has changed
  }
</code></pre>
