<p>I have a number of <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="noreferrer">Jobs</a>
running on k8s. </p>

<p>These jobs run a custom agent that copies some files and sets up the environment for a user (trusted) provided container to run.
This agent runs on the side of the user container, captures the logs, waits for the container to exit and process the generated results. </p>

<p>To achieve this, we mount Docker's socket <code>/var/run/docker.sock</code> and run as a privileged container, and from within the agent, we use <a href="https://github.com/docker/docker-py/tree/master/docker" rel="noreferrer">docker-py</a> to interact with the user container (setup, run, capture logs, terminate).</p>

<p>This works almost fine, but I'd consider it a hack. Since the user container was created by calling docker directly on a node, k8s is not aware of it's existence. This has been causing troubles since our monitoring tools interact with K8s, and don't get visibility to these stand-alone user containers. It also makes pod scheduling harder to manage, since the limits (cpu/memory) for the user container are not accounted as the requests for the pod. </p>

<p>I'm aware of <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="noreferrer">init containers</a> but these don't quite fit this use case, since we want to keep the agent running and monitoring the user container until it completes. </p>

<p><em>Is it possible for a container running on a pod, to request Kubernetes to add additional containers to the same pod the agent is running? And if so, can the agent also request Kubernetes to remove the user container at will (e.g. certain custom condition was met)?</em></p>

<p>From <a href="https://github.com/kubernetes/kubernetes/issues/37838#issuecomment-328853094" rel="nofollow noreferrer">this GitHub issue</a>, it seems that the answer is that adding or removing containers to a pod is not possible, since the container list in the pod spec is immutable. </p>
