<p>I create a <code>Namespace</code> on a kube cluster and install my app within that Namespace. The app sends regular heartbeat requests to the server and if it gets a <em>"remove yourself"</em> response from the server, it deletes itself by calling delete on the entire kube Namespace. I also give the app cluster wide access by creating a <code>ClusterRoleBinding</code>, making a <code>ServiceAccount</code> subject to the <code>ClusterRoleBinding</code> and running the pod with this <code>ServiceAccount</code>.</p>

<p>The problem is that I want to remove the <code>ClusterRoleBinding</code> as part of app's self removal process (if that's possible).  If I remove the <code>ClusterRoleBinding</code> before then the app won't be able to issue a delete on the <code>Namespace</code> so it seems like a chicken-and-egg problem. Is there a way to do this?</p>

<p>This is what I have already tried to no avail:</p>

<ul>
<li><p>Added a PreStop handler in the app container. So now when the app calls delete on the entire Namespace, kube calls this handler before killing the container. In this PreStop handler if I sleep for more than 5 sec before calling delete on <code>ClusterRoleBinding</code> I get <em>"Unauthorized"</em> response back from kubernetes.</p></li>
<li><p>This lead me to think that maybe the <code>ServiceAccount</code> that's linked to the <code>ClusterRoleBinding</code> gets deleted before the app has had a chance to delete the <code>ClusterRoleBinding</code> in the PreStop handler. So to test this, before issuing delete on the Namespace I add a finalizer to the <code>ServiceAccount</code>, then in the PreStop handler I wait for 5 sec, issue delete on the <code>ClusterRoleBinding</code> <em>(again get "Unauthorized" error back),</em> then I get the <code>ServiceAccount</code> object by name <em>(get "Unauthorized" error back)</em>, remove the finalizer from the <code>ServiceAccount</code> <em>(get error "error="finalizer doesn't exist for object ''")</em> because it can't remove the finalizer on an empty object. When I use kubectl I find that the <code>ServiceAccount</code> exists but is in <em>"Terminating"</em> state as expected with the finalizer still set.</p></li>
</ul>

<p><strong>Does kube revoke access when <code>ServiceAccount</code> is in <code>"Terminating"</code> state even though its not yet hard deleted?</strong></p>

<p><strong>Is there a way to remove the <code>ClusterRoleBinding</code> and the <code>Namespace</code> from the same process that is running in the <code>Namespace</code> that needs to be deleted? (Given that the <code>ClusterRoleBinding</code> we want to remove gives the app permission to delete the <code>Namespace</code> in the first place)</strong></p>

<p>Any help would be much appreciated!</p>

<p>The YAML definitions for <code>ClusterRoleBinding</code> and <code>ServiceAccount</code> are as below:</p>

<pre class="lang-yaml prettyprint-override"><code>### ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
    creationTimestamp: null
    name: xyz-myapp-cluster-admin
roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
subjects:
- kind: ServiceAccount
  name: xyz
  namespace: xyz-myapp

### ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
    creationTimestamp: null
    name: xyz
    namespace: xyz-myapp
</code></pre>

<p>Relevant application logs:</p>

<pre><code>time="2020-02-18T16:08:33Z" level=info msg="App instructed to remove itself"
time="2020-02-18T16:08:33Z" level=info msg="Created finalizer 'xyz.myapp.com/my-finalizer' on ServiceAccount"
time="2020-02-18T16:08:33Z" level=info msg="Called delete on Namespace"
time="2020-02-18T16:08:38Z" level=info msg="PreStop handler called"
time="2020-02-18T16:08:38Z" level=info msg="----- sleeping for 5 sec -----"
time="2020-02-18T16:08:43Z" level=info msg="Deleting ClusterRoleBinding"
time="2020-02-18T16:08:43Z" level=warning msg="Failed to delete ClusterRoleBinding" error="Unexpected error removing ClusterRolebinding: Unauthorized"
time="2020-02-18T16:08:43Z" level=warning msg="Failed to get ServiceAccount" error=Unauthorized
time="2020-02-18T16:08:43Z" level=warning msg="Failed to remove finalizer from ServiceAccount" error="finalizer 'xyz.myapp.com/my-finalizer' doesn't exist for object ''"
</code></pre>

<p>After digging through kubernetes documentation, the most reliable way I found to do this is:</p>

<ol>
<li>When the app gets a <em>"remove yourself"</em> response back from server it makes the <code>ClusterRoleBinding</code> the owner of the <code>Namespace</code> the app runs in.</li>
<li>This can be done by adding <code>ClusterRoleBinding</code> under Namespace's <code>medatadata.owerReferences</code> using patch.</li>
<li>Once <code>ClusterRoleBinding</code> is successfully added as an owner for <code>Namespace</code>, then the app can call delete on the <code>ClusterRoleBinding</code> using <code>DeletePropagationBackground</code>.</li>
</ol>

<p>Below is an example of how a patch that adds a <code>ClusterRoleBinding</code> to <code>ownerReferences</code> of a <code>Namespace</code> can be applied (in Golang).</p>

<pre class="lang-golang prettyprint-override"><code>import (
    "encoding/json"
    "k8s.io/client-go/kubernetes"
    v1 "k8s.io/api/core/v1"
    rbacv1 "k8s.io/api/rbac/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type ownerReferencePatch struct {
    Op    string                  `json:"op"`
    Path  string                  `json:"path"`
    Value []metav1.OwnerReference `json:"value"`
}

func AddClusterRoleBindingOwnerReferenceToNamespace(client kubernetes.Interface, crb *rbacv1.ClusterRoleBinding, ns *v1.Namespace) (*v1.Namespace, error) {
    patch, err := json.Marshal([]ownerReferencePatch{
        {
            Op:   "add",
            Path: "/metadata/ownerReferences",
            Value: []metav1.OwnerReference{
                {
                    APIVersion:         crb.RoleRef.APIGroup,
                    BlockOwnerDeletion: func(in bool) *bool {return &amp;in}(true),
                    Kind:               "ClusterRoleBinding",
                    Name:               crb.GetName(),
                    UID:                crb.GetUID(),
                },
            },
        },
    })
    if err != nil {
        return nil, err
    }

    return client.CoreV1().Namespaces().Patch(ns.GetName(), types.JSONPatchType, patch)
}
</code></pre>
