<h1>Purpose - What do I want to achieve?</h1>

<p>I want to access <code>systemctl</code> from inside a container running a kubernetes node (ami: running debian stretch)</p>

<h1>Working setup:</h1>

<ul>
<li><p><a href="https://github.com/kubernetes/kops/blob/master/channels/stable#L22" rel="noreferrer">Node AMI</a>: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17</p></li>
<li><p>Node Directories Mounted in the container to make <code>systemctl</code> work:</p>

<ul>
<li>/var/run/dbus</li>
<li>/run/systemd</li>
<li>/bin/systemctl</li>
<li>/etc/systemd/system</li>
</ul></li>
</ul>

<h1>Not Working setup:</h1>

<ul>
<li><p><a href="https://github.com/kubernetes/kops/blob/master/channels/stable#L26" rel="noreferrer">Node AMI</a>: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17</p></li>
<li><p>Node Directories Mounted in the container to make <code>systemctl</code> work:</p>

<ul>
<li>/var/run/dbus</li>
<li>/run/systemd</li>
<li>/bin/systemctl</li>
<li>/etc/systemd/system</li>
</ul></li>
</ul>

<h1>Debugging in an attempt to solve the problem</h1>

<p>To debug this issue with the <code>debian-stretch</code> image not supporting <code>systemctl</code> with the same mounts as <code>debian-jessie</code></p>

<p>1) I began by spinning up a nginx deployment by mounting the above mentioned volumes in it</p>

<pre><code>kubectl apply -f https://k8s.io/examples/application/deployment.yaml

kubectl exec -it nginx-deployment /bin/bash

root@nginx-deployment-788f65877d-pzzrn:/# systemctl
systemctl: error while loading shared libraries: libsystemd-shared- 
232.so: cannot open shared object file: No such file or directory
</code></pre>

<p>2) As the above issue showed the file <code>libsystemd-shared-232.so</code> not found. I found the actual path by looking into the node.</p>

<pre><code>admin@ip-10-0-20-11:~$ sudo find / -iname 'libsystemd-shared-232.so'
/lib/systemd/libsystemd-shared-232.so
</code></pre>

<p>3) Mounted the <code>/lib/systemd</code> in the nginx pod and ran the systemctl again</p>

<pre><code> kubectl exec -it nginx-deployment /bin/bash

 root@nginx-deployment-587d866f54-ghfll:/# systemctl
 systemctl: error while loading shared libraries: libcap.so.2:cannot 
 open shared object file: No such file or directory
</code></pre>

<p>4) Now the <code>systemctl</code> was failing with a new so missing error</p>

<pre><code>root@nginx-deployment-587d866f54-ghfll:/# systemctl
systemctl: error while loading shared libraries: libcap.so.2: cannot 
open shared object file: No such file or directory
</code></pre>

<p>5) To solve the above error i again searched the node for <code>libcap.so.2</code> Found it in the below path.</p>

<pre><code>admin@ip-10-0-20-11:~$ sudo find / -iname 'libcap.so.2'
/lib/x86_64-linux-gnu/libcap.so.2 
</code></pre>

<p>6) Seeing the above directory not mounted in my pod. I mounted the below path in the nginx pod.</p>

<pre><code>/lib/x86_64-linux-gnu mounted in the nginx pod(deployment)
</code></pre>

<p>7) The nginx pod  is not able to come up after adding the above mount. Getting the below error:</p>

<pre><code>$ k logs nginx-deployment-f9c5ff956-b9wn5
standard_init_linux.go:178: exec user process caused "no such file 
or directory"
</code></pre>

<p>Please suggest how to debug further. And what all mounts are required to make systemctl work from inside a container in a debian stretch environment.</p>

<p>Any pointers to take the debugging further could be helpful.</p>

<p>Rather than mounting some of the library files from the host you can just install <code>systemd</code> in the container.</p>

<pre><code>$ apt-get -y install systemd
</code></pre>

<p>Now, that won't necessarily make <code>systemctl</code> run. You will need <code>systemd</code> to be running in your container which is spawned by <code>/sbin/init</code> on your system. <code>/sbin/init</code> needs to run as root so essentially you would have to run this with the <code>privileged</code> flag in the pod or container <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#securitycontext-v1-core" rel="noreferrer">security context</a> on Kubernetes. Now, this is insecure and <a href="https://lwn.net/Articles/676831/" rel="noreferrer">there is a long history</a> about running systemd in a container where the Docker folks were mostly against it (security) and the Red Hat folks said that it was needed.</p>

<p>Nevertheless, the Red Hat folks figured out a way to <a href="https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/" rel="noreferrer">make it work without the <code>unprivileged</code> flag</a>. You need:</p>

<ul>
<li><code>/run</code> mounted as a tmpfs in the container.</li>
<li><code>/sys/fs/cgroup</code> mounted as read-only is ok.</li>
<li><code>/sys/fs/cgroup/systemd/</code> mounted as read/write. </li>
<li>Use for <code>STOPSIGNAL</code> <code>SIGRTMIN+3</code></li>
</ul>

<p>In Kubernetes you need an <a href="https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/" rel="noreferrer"><code>emptyDir</code></a> to mount a <code>tmpfs</code>. The others can be mounted as host volumes.</p>
