<p>I created a pvc, which dynamically creates a persistenvolume (using k3s with local-path) that gets used by a deployment. I am provisioning everything using terraform but encountered an error. The terraform apply enters a infinite loop while creating the pvc and pod. The pvc is in this state:</p>
<pre><code>Name:          grafana-pvc
Namespace:     default
StorageClass:  local-path
Status:        Pending
Volume:
Labels:        io.kompose.service=grafana-data
Annotations:   &lt;none&gt;
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       grafana-778c7f77c7-w7x9f
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  WaitForFirstConsumer  79s               persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled   7s (x5 over 67s)  persistentvolume-controller  waiting for pod grafana-778c7f77c7-w7x9f to be scheduled
</code></pre>
<p>and the pod is in this state:</p>
<pre><code>Name:             grafana-778c7f77c7-w7x9f
Namespace:        default
Priority:         0
Service Account:  default
Node:             &lt;none&gt;
Labels:           io.kompose.service=grafana
                  pod-template-hash=778c7f77c7
Annotations:      &lt;none&gt;
Status:           Pending
IP:
IPs:              &lt;none&gt;
Controlled By:    ReplicaSet/grafana-778c7f77c7
Containers:
  grafana:
    Image:        grafana/grafana:9.2.4
    Port:         3000/TCP
    Host Port:    0/TCP
    Environment:  &lt;none&gt;
    Mounts:
      /etc/grafana from grafana-configuration (rw)
      /var/lib/grafana from grafana-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n7cmt (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  grafana-configuration:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana-configuration
    ReadOnly:   false
  grafana-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana-pvc
    ReadOnly:   false
  kube-api-access-n7cmt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              &lt;none&gt;
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  90s   default-scheduler  0/1 nodes are available: 1 persistentvolumeclaim &quot;grafana-configuration&quot; not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  89s   default-scheduler  0/1 nodes are available: 1 persistentvolumeclaim &quot;grafana-configuration&quot; not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
</code></pre>
<p>At this stage, nothing works anymore, terraform times out and I am not able to restore the state anymore.</p>
<p>My terraform files:</p>
<p>grafana.tf:</p>
<pre><code>resource &quot;kubernetes_persistent_volume_claim&quot; &quot;grafana-configuration&quot; {
  metadata {
    name = &quot;grafana-configuration&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;grafana-configuration&quot;
    }
  }

  spec {
    access_modes = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;1Gi&quot;
      }
    }
    volume_name = &quot;grafana-configuration&quot;
  }
}

resource &quot;kubernetes_persistent_volume&quot; &quot;grafana-configuration&quot; {
  metadata {
    name = &quot;grafana-configuration&quot;
  }
  spec {
    storage_class_name = &quot;local-path&quot;
    access_modes = [&quot;ReadWriteOnce&quot;]
    capacity = {
      storage = &quot;1Gi&quot;
    }
    node_affinity {
      required {
        node_selector_term {
          match_expressions {
            key = &quot;node-role.kubernetes.io/master&quot;
            operator = &quot;In&quot;
            values = [&quot;true&quot;]
          }
        }
      } 
    }
    persistent_volume_source {
      local {
        path = &quot;/home/administrator/Metrics.Infrastructure/grafana/&quot;
      }
    }
  }
}

resource &quot;kubernetes_persistent_volume_claim&quot; &quot;grafana-pvc&quot; {
  metadata {
    name = &quot;grafana-pvc&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;grafana-data&quot;
    }
  }

  spec {
    access_modes = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;5Gi&quot;
      }
    }
  }
}

resource &quot;kubernetes_deployment&quot; &quot;grafana&quot; {
  metadata {
    name = &quot;grafana&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;grafana&quot;
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        &quot;io.kompose.service&quot; = &quot;grafana&quot;
      }
    }

    template {
      metadata {
        labels = {
          &quot;io.kompose.service&quot; = &quot;grafana&quot;
        }
      }

      spec {
        volume {
          name = &quot;grafana-configuration&quot;
          persistent_volume_claim {
            claim_name = &quot;grafana-configuration&quot;
          }
        }

        volume {
          name = &quot;grafana-data&quot;

          persistent_volume_claim {
            claim_name = &quot;grafana-pvc&quot;
          }
        }

        container {
          name  = &quot;grafana&quot;
          image = &quot;grafana/grafana:9.2.4&quot;

          port {
            container_port = 3000
          }

          volume_mount {
            name       = &quot;grafana-configuration&quot;
            mount_path = &quot;/etc/grafana&quot;
          }
          volume_mount {
            name       = &quot;grafana-data&quot;
            mount_path = &quot;/var/lib/grafana&quot;
          }
        }

        restart_policy = &quot;Always&quot;
      }
    }

    strategy {
      type = &quot;Recreate&quot;
    }
  }
}

resource &quot;kubernetes_service&quot; &quot;grafana&quot; {
    metadata {
        name = &quot;grafana&quot;

        labels = {
            &quot;io.kompose.service&quot; = &quot;grafana&quot;
        }
    }


    spec {
        port {
            port        = 3000
            target_port = 3000
            node_port = 30001
        }
        type = &quot;NodePort&quot;

        selector = {
            &quot;io.kompose.service&quot; = &quot;grafana&quot;
        }
    }
}

</code></pre>
<p>prometheus.tf:</p>
<pre><code># We need these resources so that prometheus can fetch kubernetes metrics
resource &quot;kubernetes_cluster_role&quot; &quot;prometheus-clusterrole&quot; {
  metadata {
    name = &quot;prometheus-clusterrole&quot;
  }

  rule {
    api_groups = [&quot;&quot;]
    resources  = [&quot;nodes&quot;, &quot;nodes/proxy&quot;, &quot;services&quot;, &quot;endpoints&quot;, &quot;pods&quot;]
    verbs      = [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  }
  rule {
    api_groups = [&quot;extensions&quot;]
    resources  = [&quot;ingresses&quot;]
    verbs      = [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  }
  rule {
    non_resource_urls = [&quot;/metrics&quot;]
    verbs      = [&quot;get&quot;]
  }
}

resource &quot;kubernetes_cluster_role_binding&quot; &quot;prometheus_clusterrolebinding&quot; {
  metadata {
    name = &quot;prometheus-clusterrolebinding&quot;
  }
  role_ref {
    api_group = &quot;rbac.authorization.k8s.io&quot;
    kind      = &quot;ClusterRole&quot;
    name      = &quot;prometheus-clusterrole&quot;
  }
  subject {
    kind      = &quot;ServiceAccount&quot;
    name      = &quot;default&quot;
    namespace = &quot;default&quot;
  }
}



resource &quot;kubernetes_config_map&quot; &quot;prometheus-config&quot; {
    metadata {
      name = &quot;prometheus-config&quot;
    }
    data = {
        &quot;prometheus.yml&quot; = &quot;${file(&quot;${path.module}/prometheus/prometheus.yml&quot;)}&quot;
    }
}

resource &quot;kubernetes_persistent_volume_claim&quot; &quot;prometheus_data_claim&quot; {
  metadata {
    name = &quot;prometheus-data-claim&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;prometheus-data&quot;
    }
  }

  spec {
    access_modes = [&quot;ReadWriteOnce&quot;]
    storage_class_name = &quot;local-path&quot;
    resources {
      requests = {
        storage = &quot;20Gi&quot;
      }
    }
  }
}

resource &quot;kubernetes_deployment&quot; &quot;prometheus&quot; {
  metadata {
    name = &quot;prometheus&quot;

    labels = {
      &quot;io.kompose.service&quot; = &quot;prometheus&quot;
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        &quot;io.kompose.service&quot; = &quot;prometheus&quot;
      }
    }

    template {
      metadata {
        labels = {
          &quot;io.kompose.service&quot; = &quot;prometheus&quot;
        }
      }

      spec {

        volume {
          name = &quot;prometheus-data&quot;

          persistent_volume_claim {
            claim_name = &quot;prometheus-data-claim&quot;
          }
        }

        volume {
          name = &quot;prometheus-config&quot;
          config_map {
            name = &quot;prometheus-config&quot;
          }
        }

        container {
          name  = &quot;prometheus&quot;
          image = &quot;prom/prometheus:v2.40.0&quot;
          args  = [
            &quot;--config.file=/config/prometheus.yml&quot;, 
            &quot;--storage.tsdb.path=/prometheus&quot;,
            &quot;--web.enable-lifecycle&quot;
          ]

          port {
            container_port = 9090
          }

          volume_mount {
            name       = &quot;prometheus-config&quot;
            mount_path = &quot;/config&quot;
          }

          volume_mount {
            name       = &quot;prometheus-data&quot;
            mount_path = &quot;/prometheus&quot;
          }
        }

        restart_policy = &quot;Always&quot;
      }
    }

    strategy {
      type = &quot;Recreate&quot;
    }
  }
}

resource &quot;kubernetes_service&quot; &quot;prometheus&quot; {
    metadata {
        name = &quot;prometheus&quot;

        labels = {
            &quot;io.kompose.service&quot; = &quot;prometheus&quot;
        }
    }

    spec {
        port {
            port        = 80
            target_port = 9090
            node_port = 30000
        }
        type = &quot;NodePort&quot;

        selector = {
            &quot;io.kompose.service&quot; = &quot;prometheus&quot;
        }
    }
}

</code></pre>

kaffarell
<p>In case you haven't solved your problem. I had the same issue and the fix was just a config change in the TF resource</p>
<pre><code>resource &quot;kubernetes_persistent_volume_claim_v1&quot; &quot;mypvc&quot; {
  metadata {
    ...
  }
  spec {
    ...
  }
  wait_until_bound = false
}
</code></pre>
<p><a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim#wait_until_bound" rel="nofollow noreferrer">https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim#wait_until_bound</a></p>
<p>With that TF will not wait for the PVC to be consumed, and it will be able to create the next resource which does consume the PVC</p>
