<p>I have a 5-nodes Kubernetes cluster of which 1 is the master (set up with kubeadm). When I first deployed the master node I deployed also the kubernetes dashboard so it's running on the same machine. After that I joined the other nodes to the cluster.</p>
<p>Now when I deploy a pod using a YAML file it stays in the <code>ContainerCreating</code> state. So I describe the pod and saw the machine where it was deployed. I SSH'd in the machine and checked first <code>docker ps -a</code> I could determine that the image was not started nor even was it pulled. So I looked into the kubelet logs (I didn't copy everything but this will give a pretty good idea):</p>
<pre><code>E0131 11:05:40.486422    2873 server.go:459] Kubelet needs to run as uid `0`. It is being run as 1000
W0131 11:05:40.486616    2873 server.go:469] write /proc/self/oom_score_adj: permission denied
W0131 11:05:40.486978    2873 server.go:669] No api server defined - no events will be sent to API server.
W0131 11:05:40.491423    2873 kubelet_network.go:69] Hairpin mode set to &quot;promiscuous-bridge&quot; but kubenet is not enabled, falling back to &quot;hairpin-veth&quot;
I0131 11:05:40.491498    2873 kubelet.go:477] Hairpin mode set to &quot;hairpin-veth&quot;
W0131 11:05:40.495353    2873 plugins.go:210] can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: permission denied
I0131 11:05:40.503259    2873 docker_manager.go:257] Setting dockerRoot to /var/lib/docker
I0131 11:05:40.503308    2873 docker_manager.go:260] Setting cgroupDriver to cgroupfs
I0131 11:05:40.506028    2873 server.go:770] Started kubelet v1.5.2
E0131 11:05:40.506209    2873 server.go:481] Starting health server failed: listen tcp 127.0.0.1:10248: bind: address already in use
E0131 11:05:40.506300    2873 kubelet.go:1145] Image garbage collection failed: unable to find data for container /
I0131 11:05:40.506413    2873 server.go:123] Starting to listen on 0.0.0.0:10250
W0131 11:05:40.506445    2873 kubelet.go:1224] No api server defined - no node status update will be sent.
E0131 11:05:40.507209    2873 kubelet.go:1228] error creating pods directory: mkdir /var/lib/kubelet/pods: permission denied
I0131 11:05:40.509613    2873 status_manager.go:125] Kubernetes client is nil, not starting status manager.
I0131 11:05:40.509656    2873 kubelet.go:1714] Starting kubelet main sync loop.
I0131 11:05:40.509710    2873 kubelet.go:1725] skipping pod synchronization - [error creating pods directory: mkdir /var/lib/kubelet/pods: permission denied container runtime is down]
F0131 11:05:40.509522    2873 server.go:148] listen tcp 0.0.0.0:10255: bind: address already in use
</code></pre>
<p>There are a lot of permission issues. I have no idea how to fix this. I've added root and the user account to the docker group to see if it fixes it, but it doesn't.</p>
<h1>Update</h1>
<p>Above I did a <code>kubelet logs</code> and that is why you get the uid message. When I execute <code>sudo kubelet logs</code> I get these results:</p>
<pre><code>I0201 15:36:01.386564    5082 feature_gate.go:181] feature gates: map[]
W0201 15:36:01.386861    5082 server.go:400] No API client: no api servers specified
I0201 15:36:01.386953    5082 docker.go:356] Connecting to docker on unix:///var/run/docker.sock
I0201 15:36:01.386991    5082 docker.go:376] Start docker client with request timeout=2m0s
I0201 15:36:01.401737    5082 manager.go:143] cAdvisor running in container: &quot;/user.slice&quot;
W0201 15:36:01.415664    5082 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I0201 15:36:01.431725    5082 fs.go:117] Filesystem partitions: map[/dev/mmcblk0p2:{mountpoint:/var/lib/docker/aufs major:179 minor:2 fsType:ext4 blockSize:0}]
I0201 15:36:01.440439    5082 manager.go:198] Machine: {NumCores:4 CpuFrequency:1920000 MemoryCapacity:3519315968 MachineID:a9807123b38d1f069a44f0b7588b5884 SystemUUID:03000200-0400-0500-0006-000700080009 BootID:7e71fe9b-a9d8-4921-80c7-9d09e49ed1ef Filesystems:[{Device:/dev/mmcblk0p2 Capacity:57295605760 Type:vfs Inodes:3563520 HasInodes:true}] DiskMap:map[179:0:{Name:mmcblk0 Major:179 Minor:0 Size:62545461248 Scheduler:deadline} 179:8:{Name:mmcblk0boot0 Major:179 Minor:8 Size:4194304 Scheduler:deadline} 179:16:{Name:mmcblk0boot1 Major:179 Minor:16 Size:4194304 Scheduler:deadline} 179:24:{Name:mmcblk0rpmb Major:179 Minor:24 Size:4194304 Scheduler:deadline}] NetworkDevices:[{Name:datapath MacAddress:72:36:99:b2:ba:be Speed:0 Mtu:1410} {Name:dummy0 MacAddress:ea:c7:5e:6d:29:75 Speed:0 Mtu:1500} {Name:enp1s0 MacAddress:00:07:32:3e:17:8c Speed:1000 Mtu:1500} {Name:vxlan-6784 MacAddress:5a:81:bb:f6:00:d7 Speed:0 Mtu:1500} {Name:weave MacAddress:92:64:f5:c5:57:a7 Speed:0 Mtu:1410}] Topology:[{Id:0 Memory:3519315968 Cores:[{Id:0 Threads:[0] Caches:[{Size:24576 Type:Data Level:1} {Size:32768 Type:Instruction Level:1}]} {Id:1 Threads:[1] Caches:[{Size:24576 Type:Data Level:1} {Size:32768 Type:Instruction Level:1}]} {Id:2 Threads:[2] Caches:[{Size:24576 Type:Data Level:1} {Size:32768 Type:Instruction Level:1}]} {Id:3 Threads:[3] Caches:[{Size:24576 Type:Data Level:1} {Size:32768 Type:Instruction Level:1}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0201 15:36:01.442170    5082 manager.go:204] Version: {KernelVersion:4.4.0-31-generic ContainerOsVersion:Ubuntu 16.04.1 LTS DockerVersion:1.12.3 CadvisorVersion: CadvisorRevision:}
I0201 15:36:01.444559    5082 cadvisor_linux.go:152] Failed to register cAdvisor on port 4194, retrying. Error: listen tcp :4194: bind: address already in use
W0201 15:36:01.449146    5082 container_manager_linux.go:205] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
W0201 15:36:01.449653    5082 server.go:669] No api server defined - no events will be sent to API server.
W0201 15:36:01.457574    5082 kubelet_network.go:69] Hairpin mode set to &quot;promiscuous-bridge&quot; but kubenet is not enabled, falling back to &quot;hairpin-veth&quot;
I0201 15:36:01.457658    5082 kubelet.go:477] Hairpin mode set to &quot;hairpin-veth&quot;
I0201 15:36:01.471512    5082 docker_manager.go:257] Setting dockerRoot to /var/lib/docker
I0201 15:36:01.471570    5082 docker_manager.go:260] Setting cgroupDriver to cgroupfs
I0201 15:36:01.474678    5082 server.go:770] Started kubelet v1.5.2
E0201 15:36:01.474926    5082 server.go:481] Starting health server failed: listen tcp 127.0.0.1:10248: bind: address already in use
E0201 15:36:01.475062    5082 kubelet.go:1145] Image garbage collection failed: unable to find data for container /
W0201 15:36:01.475208    5082 kubelet.go:1224] No api server defined - no node status update will be sent.
I0201 15:36:01.475702    5082 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach
I0201 15:36:01.479587    5082 server.go:123] Starting to listen on 0.0.0.0:10250
F0201 15:36:01.481605    5082 server.go:148] listen tcp 0.0.0.0:10255: bind: address already in use
</code></pre>

<p>You need to run kubelet as root (see first line of log). This a a known limitation at the moment:</p>

<p><a href="https://github.com/kubernetes/kubernetes/issues/4869" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/4869</a></p>
