<p>We recently started using istio <a href="https://istio.io" rel="noreferrer">Istio</a> to establish a service-mesh within out <a href="http://kubernetes.io" rel="noreferrer">Kubernetes</a> landscape.</p>

<p>We now have the problem that jobs and cronjobs do not terminate and keep running forever if we inject the istio <code>istio-proxy</code> sidecar container into them. The <code>istio-proxy</code> should be injected though to establish proper mTLS connections to the services the job needs to talk to and comply with our security regulations.</p>

<p>I also noticed the open issues within Istio (<a href="https://github.com/istio/istio/issues/6324" rel="noreferrer">istio/issues/6324</a>) and kubernetes (<a href="https://github.com/kubernetes/kubernetes/issues/25908" rel="noreferrer">kubernetes/issues/25908</a>), but both do not seem to provide a valid solution anytime soon.</p>

<p>At first a pre-stop hook seemed suitable to solve this issue, but there is some confusion about this conecpt itself: <a href="https://github.com/kubernetes/kubernetes/issues/55807" rel="noreferrer">kubernetes/issues/55807</a></p>

<pre><code>lifecycle:
  preStop:
    exec:
      command: 
        ...
</code></pre>

<p>Bottomline: Those hooks will not be executed if the the container successfully completed.</p>

<p>There are also some relatively new projects on GitHub trying to solve this with a dedicated controller (which I think is the most preferrable approach), but to our team they do not feel mature enough to put them right away into production:</p>

<ul>
<li><a href="https://github.com/nrmitchi/k8s-controller-sidecars" rel="noreferrer">k8s-controller-sidecars</a></li>
<li><a href="https://github.com/cropse/K8S-job-sidecar-terminator" rel="noreferrer">K8S-job-sidecar-terminator</a> </li>
</ul>

<p>In the meantime, we ourselves ended up with the following workaround that execs into the sidecar and sends a <code>SIGTERM</code> signal, but only if the main container finished successfully:</p>

<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: terminate-sidecar-example-service-account
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: terminate-sidecar-example-role
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: terminate-sidecar-example-rolebinding
subjects:
  - kind: ServiceAccount
    name: terminate-sidecar-example-service-account
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: terminate-sidecar-example-role
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: terminate-sidecar-example-cronjob
  labels:
    app: terminate-sidecar-example
spec:
  schedule: "30 2 * * *"
  jobTemplate:
    metadata:
      labels:
        app: terminate-sidecar-example
    spec:
      template:
        metadata:
          labels:
            app: terminate-sidecar-example
          annotations:
            sidecar.istio.io/inject: "true"
        spec:
          serviceAccountName: terminate-sidecar-example-service-account
          containers:
          - name: ****
            image: ****
            command:
              - "/bin/ash"
              - "-c"
            args:
              - node index.js &amp;&amp; kubectl exec -n ${POD_NAMESPACE} ${POD_NAME} -c istio-proxy -- bash -c "sleep 5 &amp;&amp; /bin/kill -s TERM 1 &amp;"
            env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
</code></pre>

<p>So, the ultimate question to all of you is: <strong>Do you know of any better workaround, solution, controller, ... that would be less hacky / more suitable to terminate the <code>istio-proxy</code> container once the main container finished its work?</strong></p>

<pre><code>- command:
  - /bin/sh
  - -c
  - |
    until curl -fsI http://localhost:15021/healthz/ready; do echo \&quot;Waiting for Sidecar...\&quot;; sleep 3; done;
    echo \&quot;Sidecar available. Running the command...\&quot;;
    &lt;YOUR_COMMAND&gt;;
    x=$(echo $?); curl -fsI -X POST http://localhost:15020/quitquitquit &amp;&amp; exit $x
</code></pre>
<p><strong>Update:</strong> sleep loop can be omitted if <code>holdApplicationUntilProxyStarts</code> is set to <code>true</code> (globally or as an annotation) starting with <a href="https://istio.io/latest/news/releases/1.7.x/announcing-1.7/change-notes/" rel="noreferrer">istio 1.7</a></p>
