<p>I create a standard nginx pod and I want to run a sidecar container, but it's not working for me in Kubernetes 1.18 - 1.20.</p>
<pre><code># kubectl alpha debug nginx --image=busybox --target=nginx
Defaulting debug container name to debugger-6wlb5.

# kubectl attach nginx -c debugger-6wlb5
If you don't see a command prompt, try pressing enter.
error: unable to upgrade connection: container debugger-6wlb5 not found in pod nginx_default 
</code></pre>
<p>I have gates feature enabled in apiserver, scheduler &amp; controller-manager. Describe on pod doesn't show errors or something:</p>
<pre><code>Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from default-token-4jhz2 (ro)
Ephemeral Containers:
  debugger-6wlb5:
    Image:        busybox
    Port:         &lt;none&gt;
    Host Port:    &lt;none&gt;
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
Conditions:
  Type              Status
</code></pre>
<p>What do I miss here?</p>

<h3>Background</h3>
<p>Unfortunately this is still <code>alfa feature</code> and it's not suitable for production. As stated in <a href="https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/" rel="nofollow noreferrer">Ephemeral Containers</a> documentation.</p>
<blockquote>
<p>This page provides an overview of ephemeral containers: a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting. You use ephemeral containers to inspect services rather than to build applications.</p>
</blockquote>
<blockquote>
<p><strong>Warning:</strong> Ephemeral containers are in early alpha state and are not suitable for production clusters.</p>
</blockquote>
<h3>Tests</h3>
<p>I have tested this on <code>Kubeadm</code> with Kubernetes 1.19. <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" rel="nofollow noreferrer">Feature Gates</a> flag was set in config files like <code>apiserver</code>, <code>scheduler</code> and <code>controller-manager</code> - <code>--feature-gates=EphemeralContainers=true</code>. Also <code>kubelet</code> was modified.
As this is <code>alpha</code> it needs specific steps to make it work.</p>
<p><strong>Option 1</strong></p>
<p>This method is described in <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container" rel="nofollow noreferrer">Debug Running Pods - Debugging with an ephemeral debug container</a>.</p>
<pre><code>$ kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
pod/ephemeral-demo created
$ kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Defaulting debug container name to debugger-hnr2w.
If you don't see a command prompt, try pressing enter.
/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    7 root      0:00 sh
   12 root      0:00 ps aux
/ # Now, you are in the debugger-hnr2w container which is in the same pod as container ephemeral-demo
/ # exit
$ 
</code></pre>
<p><em><strong>Describe pod</strong></em></p>
<pre><code>Containers:
  ephemeral-demo:
    Container ID:   docker://e24cf29efdab9fbf8180ec6c8a9539cdfbdfc490b3b4a4d5dd02d419887c8486
    Image:          k8s.gcr.io/pause:3.1
    ...
Ephemeral Containers:
  debugger-hnr2w:
    Container ID:   docker://649d020e92eedbe03d281fb2368f33ceec176eaaa340fb52af5bf59fc269c701
    Image:          busybox
    ...
</code></pre>
<p>In example above, <code>attach</code> was made by using <code>-i</code> flag. It's stated in <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#copying-a-pod-while-adding-a-new-container" rel="nofollow noreferrer">Copying a Pod while adding a new container</a>.</p>
<blockquote>
<p>The -i flag causes kubectl debug to attach to the new container by default. You can prevent this by specifying --attach=false.</p>
</blockquote>
<p><strong>Side Note</strong></p>
<p>There second part of quotation is that <code>If your session becomes disconnected you can reattach using kubectl attach.</code> however it applies to another method, not <code>Ephemeral Containers</code>.</p>
<p><strong>Option 2</strong></p>
<p>This option and all prerequisite are described in <a href="https://medium.com/01001101/ephemeral-containers-the-future-of-kubernetes-workload-debugging-c5b7ded3019f" rel="nofollow noreferrer">Ephemeral Containers — the future of Kubernetes workload debugging</a> article. Instead of <code>Deployment</code> I have used <code>nginx</code> pod.</p>
<p><code>$ kubectl run nginx --image=nginx</code>.</p>
<p>Next you have to create <code>.json</code> file with below configuration. Please change <code>metadata.name</code> to your pod name.</p>
<pre><code>{
    &quot;apiVersion&quot;: &quot;v1&quot;,
    &quot;kind&quot;: &quot;EphemeralContainers&quot;,
    &quot;metadata&quot;: {
            &quot;name&quot;: &quot;nginx&quot;
    },
    &quot;ephemeralContainers&quot;: [{
        &quot;command&quot;: [
            &quot;sh&quot;
        ],
        &quot;image&quot;: &quot;busybox:latest&quot;,
        &quot;imagePullPolicy&quot;: &quot;IfNotPresent&quot;,
        &quot;name&quot;: &quot;debugger&quot;,
        &quot;stdin&quot;: true,
        &quot;tty&quot;: true,
        &quot;terminationMessagePolicy&quot;: &quot;File&quot;
    }]
}
</code></pre>
<p><strong>IMPORTANT</strong></p>
<p>You have to apply it using <code>kubectl replace --raw</code></p>
<pre><code>$ kubectl replace --raw /api/v1/namespaces/default/pods/&lt;podName&gt;/ephemeralcontainers -f &lt;jsonFileName&gt;.json
</code></pre>
<p>Otherwise, you will get error:</p>
<pre><code>kubectl apply -f debug.json 
error: unable to recognize &quot;debug.json&quot;: no matches for kind &quot;EphemeralContainers&quot; in version &quot;v1&quot;
</code></pre>
<p>After applying proper command:</p>
<pre><code>$ kubectl replace --raw /api/v1/namespaces/default/pods/nginx/ephemeralcontainers -f debug.json
{&quot;kind&quot;:&quot;EphemeralContainers&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{&quot;name&quot;:&quot;nginx&quot;,&quot;namespace&quot;:&quot;default&quot;,&quot;selfLink&quot;:&quot;/api/v1/namespaces/default/pods/nginx/ephemeralcontainers&quot;,&quot;uid&quot;:&quot;...}
</code></pre>
<p>Use <code>kubectl describe &lt;podname&gt;</code> to verify if <code>EphemeralContainer</code> was created.</p>
<pre><code>$ kubectl describe po | grep 'Container ID' -B 2
Containers:
  nginx:
    Container ID:   docker://a410b326cdc3b95abb2edff8cdb4d7edca9498ba44b54ca6a448967596391813
--
Ephemeral Containers:
  debugger:
    Container ID:  docker://a1357c0daed0ad5664b8c838183a3eb0716339020e829077c14e7438fa5e1cf5
</code></pre>
<p>Using this method you will be able to use <code>kubectl attach</code></p>
<pre><code>$ kubectl attach -it nginx -c debugger
If you don't see a command prompt, try pressing enter.
/ # 
</code></pre>
<h3>Conclusion</h3>
<p><code>Ephemeral Container</code> is a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting. One you will terminate session, you will not be able to connect again.</p>
<p>You will get error:</p>
<pre><code>$ kubectl attach -it nginx -c debugger
If you don't see a command prompt, try pressing enter.
error: unable to upgrade connection: container debugger not found in pod nginx_default
</code></pre>
<p>It might be changed in the future but based on my tests, you can connect to this container only once.</p>
