<p>How do I force delete Namespaces stuck in Terminating?</p>

<h2>Steps to recreate:</h2>

<ol>
<li>Apply this YAML</li>
</ol>

<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
  finalizers:
    - foregroundDeletion
</code></pre>

<ol start="2">
<li><p><code>kubectl delete ns delete-me</code></p></li>
<li><p>It is not possible to delete <code>delete-me</code>.</p></li>
</ol>

<p>The only workaround I've found is to destroy and recreate the entire cluster.</p>

<h2>Things I've tried:</h2>

<p>None of these work or modify the Namespace. After any of these the problematic finalizer still exists.</p>

<h3>Edit the YAML and <code>kubectl apply</code></h3>

<p>Apply:</p>

<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
  finalizers:
</code></pre>

<pre><code>$ kubectl apply -f tmp.yaml 

namespace/delete-me configured
</code></pre>

<p>The command finishes with no error, but the Namespace is not udpated.</p>

<p>The below YAML has the same result:</p>

<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
</code></pre>

<h3><code>kubectl edit</code></h3>

<p><code>kubectl edit ns delete-me</code>, and remove the finalizer. Ditto removing the list entirely. Ditto removing <code>spec</code>. Ditto replacing <code>finalizers</code> with an empty list.</p>

<pre><code>$ kubectl edit ns delete-me 

namespace/delete-me edited
</code></pre>

<p>This shows no error message but does not update the Namespace. <code>kubectl edit</code>ing the object again shows the finalizer still there.</p>

<h3><code>kubectl proxy &amp;</code></h3>

<ul>
<li><code>kubectl proxy &amp;</code></li>
<li><code>curl -k -H "Content-Type: application/yaml" -X PUT --data-binary @tmp.yaml http://127.0.0.1:8001/api/v1/namespaces/delete-me/finalize</code></li>
</ul>

<p>As above, this exits successfully but does nothing.</p>

<h3>Force Delete</h3>

<p><code>kubectl delete ns delete-me --force --grace-period=0</code></p>

<p>This actually results in an error:</p>

<pre><code>warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "delete-me": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
</code></pre>

<p>However, it doesn't actually do anything.</p>

<h3>Wait a long time</h3>

<p>In the test cluster I set up to debug this issue, I've been waiting over a week. Even if the Namespace might eventually decide to be deleted, I need it to be deleted faster than a week.</p>

<h3>Make sure the Namespace is empty</h3>

<p>The Namespace is empty.</p>

<pre><code>$ kubectl get -n delete-me all

No resources found.
</code></pre>

<h3><code>etcdctl</code></h3>

<pre><code>$ etcdctl --endpoint=http://127.0.0.1:8001 rm /namespaces/delete-me

Error:  0:  () [0]
</code></pre>

<p>I'm pretty sure that's an error, but I have no idea how to interpret that. It also doesn't work. Also tried with <code>--dir</code> and <code>-r</code>.</p>

<h3><code>ctron/kill-kube-ns</code></h3>

<p>There is a <a href="https://github.com/ctron/kill-kube-ns" rel="noreferrer">script for force deleting Namespaces</a>. This also does not work.</p>

<pre><code>$ ./kill-kube-ns delete-me

Killed namespace: delete-me

$ kubectl get ns delete-me 

NAME        STATUS        AGE
delete-me   Terminating   1h
</code></pre>

<h3><code>POST</code>ing the edited resource to /finalize</h3>

<p><a href="https://gist.github.com/willbeason/be90dac4c9d1b2a26b8c0a10ac1e50dc" rel="noreferrer">Returns a 405</a>. I'm not sure if this is the canonical way to POST to /finalize though.</p>

<h2>Links</h2>

<p><a href="https://github.com/kubernetes/kubernetes/issues/19317" rel="noreferrer">This</a>
<a href="https://github.com/kubernetes/kubernetes/issues/73186" rel="noreferrer">appears</a>
<a href="https://github.com/kubernetes/kubernetes/issues/60807" rel="noreferrer">to</a>
<a href="https://stackoverflow.com/questions/52369247/namespace-stucked-as-terminating-how-i-removed-it">be</a>
<a href="https://stackoverflow.com/questions/52954174/kubernetes-namespaces-stuck-in-terminating-status">a</a>
<a href="https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.1/troubleshoot/ns_terminating.html" rel="noreferrer">recurring</a>
<a href="https://nasermirzaei89.net/2019/01/27/delete-namespace-stuck-at-terminating-state/" rel="noreferrer">problem</a>
<a href="https://unofficial-kubernetes.readthedocs.io/en/latest/tasks/administer-cluster/namespaces/" rel="noreferrer">and</a>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1540779" rel="noreferrer">none</a>
<a href="https://linuxhelp4u.blogspot.com/2019/01/kubernetes-remove-namespace-stuck-in.html" rel="noreferrer">of</a>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1541350" rel="noreferrer">these</a>
<a href="https://github.com/kubernetes/kubernetes/issues/37554" rel="noreferrer">resources</a>
<a href="https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/" rel="noreferrer">helped</a>.</p>

<p><a href="https://github.com/kubernetes/kubernetes/issues/77086" rel="noreferrer">Kubernetes bug</a></p>

Will Beason
<p>The <code>kubectl proxy</code> try is almost correct, but not quite. It's possible using JSON instead of YAML does the trick, but I'm not certain.</p>

<p>The JSON with an empty finalizers list:</p>

<pre><code>~$ cat ns.json

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "delete-me"
  },
  "spec": {
    "finalizers": []
  }
}
</code></pre>

<p>Use <code>curl</code> to <code>PUT</code> the object without the problematic finalizer.</p>

<pre><code>~$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8007/api/v1/namespaces/delete-me/finalize

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "delete-me",
    "selfLink": "/api/v1/namespaces/delete-me/finalize",
    "uid": "0df02f91-6782-11e9-8beb-42010a800137",
    "resourceVersion": "39047",
    "creationTimestamp": "2019-04-25T17:46:28Z",
    "deletionTimestamp": "2019-04-25T17:46:31Z",
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"delete-me\"},\"spec\":{\"finalizers\":[\"foregroundDeletion\"]}}\n"
    }
  },
  "spec": {

  },
  "status": {
    "phase": "Terminating"
  }
}
</code></pre>

<p>The Namespace is deleted!</p>

<pre><code>~$ kubectl get ns delete-me

Error from server (NotFound): namespaces "delete-me" not found
</code></pre>
