title: K8s namespace terminating for 2 days
body: <p>I was attempting to remove fluxcd from a kubernetes cluster and it is stuck at deleting the namespace 'flux-system'. I've tried removing 'kubernetes' from the finalizers but getting a 404 when using curl to the api and 'invalid object' when trying '&quot;/api/v1/namespaces/flux-system/finalize&quot; -f flux-system.json'.</p>
  <pre><code>kubectl get ns flux-system -ojson | jq '.spec.finalizers = []' | kubectl replace --raw &quot;/api/v1/namespaces/flux-system/finalize&quot; -f -
  Error from server (Conflict): Operation cannot be fulfilled on namespaces &quot;flux-system&quot;: StorageError: invalid object, Code: 4, Key: /registry/namespaces/flux-system, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 6c59b2a0-d6b4-42de-bd2a-d148ad22ec19, UID in object meta:
  </code></pre>
  <pre><code>curl -v -k -H &quot;Content-type: application/json&quot; -X PUT --data-binary @flux.json http://127.0.0.1:8001/k8s/clusters/c-5hqd5/v1/flux-system/finalize
   *   Trying 127.0.0.1...
   * TCP_NODELAY set
   * Connected to 127.0.0.1 (127.0.0.1) port 8001 (#0)
  &gt; PUT /k8s/clusters/c-5hqd5/v1/flux-system/finalize HTTP/1.1
  &gt; Host: 127.0.0.1:8001
  &gt; User-Agent: curl/7.64.1
  &gt; Accept: */*
  &gt; Content-type: application/json
  &gt; Content-Length: 3180
  &gt; Expect: 100-continue
  &gt;
   * Done waiting for 100-continue
   * We are completely uploaded and fine
  &lt; HTTP/1.1 100 Continue
  &lt; HTTP/1.1 404 Not Found
  &lt; Content-Length: 0
  &lt; Date: Fri, 27 May 2022 15:34:58 GMT
  &lt; Strict-Transport-Security: max-age=15724800; includeSubDomains
  &lt; X-Content-Type-Options: nosniff
  &lt; X-Content-Type-Options: nosniff
  &lt;
  Connection #0 to host 127.0.0.1 left intact
  Closing connection 0
  </code></pre>
  <ul>
  <li>Rancher v2.5.7</li>
  <li>Kubernets v1.20.11</li>
  </ul>
  <blockquote>
  <p>kubectl edit namespace flux-system</p>
  </blockquote>
  <pre><code>apiVersion: v1
  kind: Namespace
  metadata:
    annotations:
      cattle.io/status: '{&quot;Conditions&quot;:[{&quot;Type&quot;:&quot;ResourceQuotaInit&quot;,&quot;Status&quot;:&quot;True&quot;,&quot;Message&quot;:&quot;&quot;,&quot;LastUpdateTime&quot;:&quot;2022-05-20T17:33:54Z&quot;},{&quot;Type&quot;:&quot;InitialRolesPopulated&quot;,&quot;Status&quot;:&quot;True&quot;,&quot;Message&quot;:&quot;&quot;,&quot;LastUpdateTime&quot;:&quot;2022-05-20T17:33:54Z&quot;}]}'
      field.cattle.io/projectId: c-5hqd5:p-4hbs6
      lifecycle.cattle.io/create.namespace-auth: &quot;true&quot;
    creationTimestamp: &quot;2022-05-20T17:33:53Z&quot;
    deletionTimestamp: &quot;2022-05-20T21:46:36Z&quot;
    labels:
      app.kubernetes.io/instance: flux-system
      app.kubernetes.io/part-of: flux
      app.kubernetes.io/version: v0.28.5
      field.cattle.io/projectId: p-4hbs6
      kustomize.toolkit.fluxcd.io/name: flux-system
      kustomize.toolkit.fluxcd.io/namespace: flux-system
      pod-security.kubernetes.io/warn: restricted
      pod-security.kubernetes.io/warn-version: latest
    name: flux-system
    resourceVersion: &quot;147598092&quot;
    uid: 6c59b2a0-d6b4-42de-bd2a-d148ad22ec19
  spec:
    finalizers:
    - kubernetes
  status:
    conditions:
    - lastTransitionTime: &quot;2022-05-20T21:46:41Z&quot;
      message: 'Discovery failed for some groups, 2 failing: unable to retrieve the
        complete list of server APIs: flowcontrol.apiserver.k8s.io/v1beta1: the server
        could not find the requested resource, node.k8s.io/v1: the server could not
        find the requested resource'
      reason: DiscoveryFailed
      status: &quot;True&quot;
      type: NamespaceDeletionDiscoveryFailure
    - lastTransitionTime: &quot;2022-05-20T21:46:42Z&quot;
      message: All legacy kube types successfully parsed
      reason: ParsedGroupVersions
      status: &quot;False&quot;
      type: NamespaceDeletionGroupVersionParsingFailure
    - lastTransitionTime: &quot;2022-05-20T21:46:42Z&quot;
      message: All content successfully deleted, may be waiting on finalization
      reason: ContentDeleted
      status: &quot;False&quot;
      type: NamespaceDeletionContentFailure
    - lastTransitionTime: &quot;2022-05-20T21:46:56Z&quot;
      message: All content successfully removed
      reason: ContentRemoved
      status: &quot;False&quot;
      type: NamespaceContentRemaining
    - lastTransitionTime: &quot;2022-05-20T21:46:42Z&quot;
      message: All content-preserving finalizers finished
      reason: ContentHasNoFinalizers
      status: &quot;False&quot;
      type: NamespaceFinalizersRemaining
    phase: Terminating
  </code></pre>