<p><strong>Goal</strong>:  Restart Docker daemon on GKE</p>
<p><strong>Issue</strong>: Cannot connect to bus</p>
<p><strong>Background</strong>
While on Google Kubernetes Engine (GKE), I am attempting to restart the host node's Docker daemon in order to enable the <a href="https://docs.nvidia.com/datacenter/cloud-native/kubernetes/dcgme2e.html" rel="nofollow noreferrer">Nvidia GPU Telemetry for Kubernetes</a> on nodes that have a GPU.  I have correctly isolated just the GPU nodes properly, and I am able to run every command on the host node by having a <code>DaemonSet</code> run an <code>initContainer</code> following the <a href="https://github.com/GoogleCloudPlatform/solutions-gke-init-daemonsets-tutorial" rel="nofollow noreferrer">Automatically bootstrapping Kubernetes Engine nodes with daemonSets</a> guide.</p>
<p>During runtime, however, the following pod does not allow me to connect to the Docker daemon:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: debug
  namespace: gpu-monitoring
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: cloud.google.com/gke-accelerator
            operator: Exists
  containers:
  - command:
    - sleep
    - &quot;86400&quot;
    env:
    - name: ROOT_MOUNT_DIR
      value: /root
    image: docker.io/ubuntu:18.04
    imagePullPolicy: IfNotPresent
    name: node-initializer
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /root
      name: root
    - mountPath: /scripts
      name: entrypoint
    - mountPath: /run
      name: run
  volumes:
  - hostPath:
      path: /
      type: &quot;&quot;
    name: root
  - configMap:
      defaultMode: 484
      name: nvidia-container-toolkit-installer-entrypoint
    name: entrypoint
  - hostPath:
      path: /run
      type: &quot;&quot;
    name: run
</code></pre>
<p>The user is <code>0</code>, while the users present in <code>/run/user</code> are <code>1003</code>, and <code>1002</code>.</p>
<p>In order to verify connectivity and interactions with the root Kubernetes (k8s) node, the following is run:</p>
<pre class="lang-sh prettyprint-override"><code>root@debug:/# chroot &quot;${ROOT_MOUNT_DIR}&quot; ps aux

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 226124  9816 ?        Ss   Oct13   0:27 /sbin/init
</code></pre>
<p><strong>The Issues</strong></p>
<p><em>Both images</em></p>
<p>When attempting to interact with the underlying Kubernetes (k8s) node to restart the Docker daemon, I get the following:</p>
<pre><code>root@debug:/# ls /run/dbus

system_bus_socket

root@debug:/# ROOT_MOUNT_DIR=&quot;${ROOT_MOUNT_DIR:-/root}&quot;
root@debug:/# chroot &quot;${ROOT_MOUNT_DIR}&quot; systemctl status docker

Failed to connect to bus: No data available
</code></pre>
<p>When attempting to start dbus on the host node:</p>
<pre class="lang-sh prettyprint-override"><code>root@debug:/# export XDG_RUNTIME_DIR=/run/user/`id -u`
root@debug:/# export DBUS_SESSION_BUS_ADDRESS=&quot;unix:path=${XDG_RUNTIME_DIR}/bus&quot;
root@debug:/# chroot &quot;${ROOT_MOUNT_DIR}&quot; /etc/init.d/dbus start

Failed to connect to bus: No data available
</code></pre>
<p><em>Image: solita/ubuntu-systemd</em></p>
<p>When trying to run commands using the same k8s pod config, except inside the <code>solita/ubuntu-systemd</code> image, the following are the results:</p>
<pre class="lang-sh prettyprint-override"><code>root@debug:/# /etc/init.d/dbus start
[....] Starting dbus (via systemctl): dbus.serviceRunning in chroot, ignoring request: start
. ok 
</code></pre>
<p><strong>Configuration Variations Attempted</strong>
I have tried to change the following, in pretty much every combination, to no avail:</p>
<ul>
<li>Image to <code>docker.io/solita/ubuntu-systemd:18.04</code></li>
<li>Add <code>shareProcessNamespace: true</code></li>
<li>Add the following mounts:  <code>/dev</code>, <code>/proc</code>, <code>/sys</code></li>
<li>Restrict <code>/run</code> to <code>/run/dbus</code>, and <code>/run/systemd</code></li>
</ul>

Eric Meadows
<p>So the answer is a weird workaround that was not fully expected.  In order to restart the Docker daemon, first punch a firewall hole for pods to connect to the host node.  Next, use <code>gcloud compute ssh</code>, and ssh into the node and restart via a remote ssh command:</p>
<pre class="lang-sh prettyprint-override"><code>apt-get update
apt-get install -y \
  apt-transport-https \
  curl \
  gnupg \
  lsb-release \
  ssh

export CLOUD_SDK_REPO=&quot;cloud-sdk-$(lsb_release -c -s)&quot;
echo &quot;deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main&quot; | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update
apt-get install -y google-cloud-sdk

CLUSTER_NAME=&quot;$(curl -sS http://metadata/computeMetadata/v1/instance/attributes/cluster-name -H &quot;Metadata-Flavor: Google&quot;)&quot;
NODE_NAME=&quot;$(curl -sS http://metadata.google.internal/computeMetadata/v1/instance/name -H 'Metadata-Flavor: Google')&quot;
FULL_ZONE=&quot;$(curl -sS http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google' | awk -F  &quot;/&quot; '{print $4}')&quot;
MAIN_ZONE=$(echo $FULL_ZONE | sed 's/\(.*\)-.*/\1/')

gcloud compute ssh \
  --internal-ip $NODE_NAME \
  --zone=$FULL_ZONE \
  -- &quot;sudo systemctl restart docker&quot;

</code></pre>
