<p>I am experiencing an issue using the helm_release resource in Terraform.</p>
<p>I basically deployed a <code>kube-prometheus-stack</code> that includes many k8s resources and that works smoothly.</p>
<p>The problem arose when I tried to destroy (or remove) this part, since Helm does not delete all resources (it probably has to do with some garbage collection rule that keeps them up and running after delete) when uninstalls the chart. That means I end-up with:</p>
<ul>
<li>chart uninstalled</li>
<li>resources still up and running</li>
<li>having to manually go there and remove everything, otherwise if I re-create the thing I get plenty of duplicates</li>
</ul>
<p>I previously asked a question (that now I am closing) related to understanding if it was a problem with Helm (and is not, by design it deletes everything when it can, I am not sure if in the chart something can be done, but anyway I am assuming it won't be done now and quickly) now I would like to ask if somebody has ab idea on how I can manage this directly from Terraform.</p>
<p>Is there something I can use to, for instance, run a <code>kubectl delete</code> command on the labelled resources (or maybe the whole namespace) when the <code>helm_release</code> resource gets destroyed?</p>
<p>Note: I am not adding any code, since this has nothing to do with the code, but more about finding some hook or hack to run a cleanup only after destroy.</p>
<p>p.s.: I also tried to exploit Terraform cloud hooks post-apply, but I would prefer to solve this within depending on Terraform Cloud and, anyway, I didn't manage to create a dependency on whether the <code>helm_release</code> had been destroyed.</p>

Bertone
<p>In you need to solve this problem directly from Terraform, you could consider using a <a href="https://developer.hashicorp.com/terraform/language/resources/provisioners/null_resource" rel="nofollow noreferrer"><code>null_resource</code></a> with a <a href="https://developer.hashicorp.com/terraform/language/resources/provisioners/local-exec" rel="nofollow noreferrer"><code>local-exec</code> provisioner</a> that is triggered when the <a href="https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release" rel="nofollow noreferrer"><code>helm_release</code> resource</a> gets destroyed.<br />
The <code>local-exec</code> provisioner invokes a local executable, in this case <code>kubectl</code>, after a resource is destroyed.</p>
<p>Here is a brief example of how this could work:</p>
<pre><code>resource &quot;helm_release&quot; &quot;example&quot; {
  name       = &quot;example&quot;
  namespace  = &quot;default&quot;
  chart      = &quot;stable/kube-prometheus-stack&quot;
  # add your values here...
}

resource &quot;null_resource&quot; &quot;cleanup&quot; {
  triggers = {
    helm_release_id = helm_release.example.id
  }

  provisioner &quot;local-exec&quot; {
    when    = destroy
    command = &quot;kubectl delete namespace ${helm_release.example.namespace}&quot;
  }
}
</code></pre>
<p>The above script will run <code>kubectl delete namespace</code> command on the namespace after the <code>helm_release</code> resource is destroyed.</p>
<p><strong>Do test that carefully</strong>: deleting the <em>entire</em> namespace, not just the resources created by the Helm chart, is not a casual operation!<br />
If there are other resources in the namespace that you do not want to delete, you will need to modify the <code>kubectl</code> command to delete only the resources you want.</p>
<p>And note that you would need to have <code>kubectl</code> configured on the machine running Terraform and it needs to have appropriate permissions to delete resources in your Kubernetes cluster.</p>
<p>Also, this <code>null_resource</code> will not get created until after the <code>helm_release</code> is created, due to the dependency in the <code>triggers</code> block. So, if the <code>helm_release</code> creation fails for some reason, the <code>null_resource</code> and its provisioners will not be triggered.</p>
<hr />
<blockquote>
<p>Unfortunately, I am using Terraform Cloud in a CI/CD pipe, therefore I won't be able to exploit the local-exec. But the answer is close to what I was looking for and since I didn't specify about Terraform Cloud is actually right.<br />
Do you have any other idea?</p>
</blockquote>
<p>The <code>local-exec</code> provisioner indeed cannot be used in the Terraform Cloud as it does not support running arbitrary commands on the host running Terraform.</p>
<h2>Kubernetes Provider lifecycle management</h2>
<p>An alternative solution in this context would be to use <a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs" rel="nofollow noreferrer">Kubernetes providers in Terraform</a> to manage lifecycle of the resources that are left behind.</p>
<p>For example, let's say your Helm chart leaves behind a <code>PersistentVolumeClaim</code> resource. You could manage this using the Kubernetes provider in Terraform:</p>
<pre><code>provider &quot;kubernetes&quot; {
  # configuration for your Kubernetes cluster
}

resource &quot;helm_release&quot; &quot;example&quot; {
  name       = &quot;example&quot;
  namespace  = &quot;default&quot;
  chart      = &quot;stable/kube-prometheus-stack&quot;
  # add your values
}

data &quot;kubernetes_persistent_volume_claim&quot; &quot;pvc&quot; {
  metadata {
    name      = &quot;my-pvc&quot;
    namespace = helm_release.example.namespace
  }
}

resource &quot;kubernetes_persistent_volume_claim&quot; &quot;pvc&quot; {
  depends_on = [helm_release.example]

  metadata {
    name      = data.kubernetes_persistent_volume_claim.pvc.metadata.0.name
    namespace = data.kubernetes_persistent_volume_claim.pvc.metadata.0.namespace
  }

  spec {
    access_modes = data.kubernetes_persistent_volume_claim.pvc.spec.0.access_modes
    resources {
      requests = {
        storage = data.kubernetes_persistent_volume_claim.pvc.spec.0.resources.0.requests[&quot;storage&quot;]
      }
    }

    volume_name = data.kubernetes_persistent_volume_claim.pvc.spec.0.volume_name
  }
}
</code></pre>
<p>In this example, the <code>kubernetes_persistent_volume_claim</code> resource will delete the PVC when the Terraform stack is destroyed.</p>
<p>You would have to do this for every type of resource that is left behind, so it can be a bit tedious, but it is an option.</p>
<h2>Kubernetes Provider for Job or a script</h2>
<p>Another approach would be using the Kubernetes provider to call a Kubernetes Job or a script that cleans up the resources left behind:</p>
<pre><code>provider &quot;kubernetes&quot; {
  # configuration for your Kubernetes cluster goes here
}

resource &quot;helm_release&quot; &quot;example&quot; {
  name       = &quot;example&quot;
  namespace  = &quot;default&quot;
  chart      = &quot;stable/kube-prometheus-stack&quot;
  # add your values here...
}

resource &quot;kubernetes_job&quot; &quot;cleanup&quot; {
  metadata {
    name      = &quot;cleanup-job&quot;
    namespace = helm_release.example.namespace
  }

  spec {
    template {
      metadata {}
      spec {
        container {
          name    = &quot;cleanup&quot;
          image   = &quot;appropriate/curl&quot; # or any image that has kubectl or equivalent tool
          command = [&quot;sh&quot;, &quot;-c&quot;, &quot;kubectl delete ...&quot;] # replace ... with the actual cleanup commands
        }
        
        restart_policy = &quot;Never&quot;
      }
    }

    backoff_limit = 4
  }

  depends_on = [helm_release.example]
}
</code></pre>
<p>In this second example, the <code>kubernetes_job</code> resource is triggered when the <code>helm_release</code> resource is created, running a cleanup script. The cleanup script could delete any resources that are left behind by the Helm chart.</p>
<p>Remember that in both cases, the Kubernetes provider needs to be properly configured and that the Kubernetes cluster permissions must allow the actions you are trying to perform.</p>
<hr />
<p>Regarding the second example, the OP asks if it is possible for the <code>kubernetes_job</code> to be triggered automatically when the <code>helm_release</code> resource gets destroyed.</p>
<p>Unfortunately, Terraform's built-in resources and providers do not provide a direct way to execute something only upon the destruction of another resource. The <code>provisioner</code> block is a way to do this, but as we discussed, it is not suitable for Terraform Cloud and cannot be used with the Kubernetes provider directly.</p>
<p>As an indirect solution, you can create a Kubernetes job that is configured to delete the resources as soon as it is launched, and then use a <a href="https://developer.hashicorp.com/terraform/language/meta-arguments/depends_on" rel="nofollow noreferrer"><code>depends_on</code> reference</a> to the <code>helm_release</code> in the job's configuration. That way, whenever the Helm release is created, the job will be launched as well. When you run <code>terraform destroy</code>, the Helm release will be destroyed and the job will be launched once more, thereby cleaning up the resources.</p>
<p>However, this approach is not perfect because it will also run the job when the resources are first created, not only when they are destroyed.</p>
<p>To address that,  you could write your cleanup script such that it is idempotent and will not fail or cause any negative side effects if it is run when it is not necessary (i.e., upon creation of the Helm release).<br />
For example, your script could first check if the resources it is supposed to clean up actually exist before attempting to delete them:</p>
<pre><code>provider &quot;kubernetes&quot; {
  # configuration for your Kubernetes cluster goes here
}

resource &quot;helm_release&quot; &quot;example&quot; {
  name       = &quot;example&quot;
  namespace  = &quot;default&quot;
  chart      = &quot;stable/kube-prometheus-stack&quot;
  # add your values here...
}

resource &quot;kubernetes_job&quot; &quot;cleanup&quot; {
  depends_on = [helm_release.example]

  metadata {
    name      = &quot;cleanup-job&quot;
    namespace = helm_release.example.namespace
  }

  spec {
    template {
      metadata {}
      spec {
        container {
          name    = &quot;cleanup&quot;
          image   = &quot;appropriate/curl&quot; # or any image that has kubectl or equivalent tool
          command = [&quot;sh&quot;, &quot;-c&quot;, 
                     &quot;if kubectl get &lt;resource&gt; &lt;name&gt;; then kubectl delete &lt;resource&gt; &lt;name&gt;; fi&quot;]
                     # replace &lt;resource&gt; and &lt;name&gt; with the actual resource and name
        }

        restart_policy = &quot;Never&quot;
      }
    }

    backoff_limit = 4
  }
}
</code></pre>
<p>In this example, the command checks if a specific Kubernetes resource exists before attempting to delete it. That way, the job can be safely run whether the Helm release is being created or destroyed, and the cleanup will only occur if the resource exists.</p>
<p>Do replace <code>&lt;resource&gt;</code> and <code>&lt;name&gt;</code> with the actual resource and name of the resource you wish to check and potentially delete.</p>
