{"action":"created","issue":{"url":"https://api.github.com/repos/luxas/kubernetes-on-arm/issues/96","repository_url":"https://api.github.com/repos/luxas/kubernetes-on-arm","labels_url":"https://api.github.com/repos/luxas/kubernetes-on-arm/issues/96/labels{/name}","comments_url":"https://api.github.com/repos/luxas/kubernetes-on-arm/issues/96/comments","events_url":"https://api.github.com/repos/luxas/kubernetes-on-arm/issues/96/events","html_url":"https://github.com/luxas/kubernetes-on-arm/issues/96","id":150911396,"number":96,"title":"master does not have anything running on 8080","user":{"login":"richtong","id":1782087,"avatar_url":"https://avatars.githubusercontent.com/u/1782087?v=3","gravatar_id":"","url":"https://api.github.com/users/richtong","html_url":"https://github.com/richtong","followers_url":"https://api.github.com/users/richtong/followers","following_url":"https://api.github.com/users/richtong/following{/other_user}","gists_url":"https://api.github.com/users/richtong/gists{/gist_id}","starred_url":"https://api.github.com/users/richtong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/richtong/subscriptions","organizations_url":"https://api.github.com/users/richtong/orgs","repos_url":"https://api.github.com/users/richtong/repos","events_url":"https://api.github.com/users/richtong/events{/privacy}","received_events_url":"https://api.github.com/users/richtong/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-04-25T16:04:43Z","updated_at":"2016-05-15T22:23:36Z","closed_at":null,"body":"OK, Luxas what's the best way to debug this? I have gone back to the original RPI1. It is running hypriot 201603. I have done a kube-config install and then a `kube-config enable-master` and I see one container called kubernetesonarm/hyperkube start up. Yeah!\r\n\r\nBut when I go to the second pi and try `kube-config enable-worker 10.0.1.152` for instance, I get a master not up. In looking at your code, it is checkout for 10.0.1.152:8080 and there is nothing on that port. \r\n\r\nIn looking at your start-master I checked to see if the services system-docker, etcd, flannel and k8s-master all said they were started but when I do a curl 127.0.0.1:8080 even on master it says connection refused.\r\n\r\nWhat's the best way to debug this. Sorry I'm such a noob on kubernetes. I keep getting this /var/log/syslog message:\r\n\r\n```\r\nApr 25 09:19:25 alex0 sh[2183]: I0425 16:19:25.227805    2202 kubelet.go:1134] Attempting to register node 10.0.1.152\r\nApr 25 09:19:25 alex0 sh[2183]: I0425 16:19:25.245395    2202 kubelet.go:1137] Unable to register 10.0.1.152 with the apiserver: Post http://localhost:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: connection refused\r\nApr 25 09:19:27 alex0 sh[2183]: I0425 16:19:27.562262    2202 kubelet.go:2365] skipping pod synchronization - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset]]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/luxas/kubernetes-on-arm/issues/comments/219314514","html_url":"https://github.com/luxas/kubernetes-on-arm/issues/96#issuecomment-219314514","issue_url":"https://api.github.com/repos/luxas/kubernetes-on-arm/issues/96","id":219314514,"user":{"login":"johan-wallermark","id":4752604,"avatar_url":"https://avatars.githubusercontent.com/u/4752604?v=3","gravatar_id":"","url":"https://api.github.com/users/johan-wallermark","html_url":"https://github.com/johan-wallermark","followers_url":"https://api.github.com/users/johan-wallermark/followers","following_url":"https://api.github.com/users/johan-wallermark/following{/other_user}","gists_url":"https://api.github.com/users/johan-wallermark/gists{/gist_id}","starred_url":"https://api.github.com/users/johan-wallermark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johan-wallermark/subscriptions","organizations_url":"https://api.github.com/users/johan-wallermark/orgs","repos_url":"https://api.github.com/users/johan-wallermark/repos","events_url":"https://api.github.com/users/johan-wallermark/events{/privacy}","received_events_url":"https://api.github.com/users/johan-wallermark/received_events","type":"User","site_admin":false},"created_at":"2016-05-15T22:23:36Z","updated_at":"2016-05-15T22:23:36Z","body":"I ran into the same problem. I'm not really sure where to being troubleshooting this so any suggestions would be appreciated. Here's some information.\r\n\r\nI did a clean install using the latest in dev.\r\n```\r\nsdcard/write.sh /dev/sdf rpi-3 archlinux kube-systemd\r\nConfig wlan\r\nkube-config install\r\nkube-config enable-master\r\n```\r\n(Don't mind the hostname)\r\n\r\n```\r\n[root@rpi-slave01 ~]# kube-config info\r\nArchitecture: armv7l\r\nKernel: Linux 4.4.10\r\nCPU: 4 cores x 1200 MHz\r\n\r\nUsed RAM Memory: 73 MiB\r\nRAM Memory: 922 MiB\r\n\r\nUsed disk space: 2.7GB (2733040 KB)\r\nFree disk space: 12GB (11624336 KB)\r\n\r\nSD Card/deb package was built: 15-05-2016 06:56\r\n\r\nkubernetes-on-arm:\r\nLatest commit: 290cef1\r\nVersion: 0.7.0\r\n\r\nsystemd version: v229\r\ndocker version: v1.10.0\r\nkubernetes client version: v1.2.0\r\n\r\n[root@rpi-slave01 ~]# mount\r\n/dev/mmcblk0p2 on / type ext4 (rw,relatime,data=ordered)\r\ndevtmpfs on /dev type devtmpfs (rw,relatime,size=467888k,nr_inodes=116972,mode=755)\r\nsysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)\r\nproc on /proc type proc (rw,nosuid,nodev,noexec,relatime)\r\nsecurityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)\r\ntmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)\r\ndevpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)\r\ntmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)\r\ntmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)\r\ncgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)\r\ncgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)\r\ncgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)\r\ncgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)\r\ncgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)\r\ncgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)\r\ncgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)\r\ncgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)\r\ncgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)\r\nsystemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=24,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)\r\nmqueue on /dev/mqueue type mqueue (rw,relatime)\r\ndebugfs on /sys/kernel/debug type debugfs (rw,relatime)\r\ntmpfs on /tmp type tmpfs (rw)\r\nconfigfs on /sys/kernel/config type configfs (rw,relatime)\r\n/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)\r\noverlay on /var/lib/docker/overlay/92abe79255bb7ad059f4bbb01c0de7419b6a1bf69f2414348c43dc3967a92e95/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/d8c7bdb962528d534a8a61af449b87eba8a630397a5553db510506d6d9eb496d/root,upperdir=/var/lib/docker/overlay/92abe79255bb7ad059f4bbb01c0de7419b6a1bf69f2414348c43dc3967a92e95/upper,workdir=/var/lib/docker/overlay/92abe79255bb7ad059f4bbb01c0de7419b6a1bf69f2414348c43dc3967a92e95/work)\r\nnsfs on /run/docker/netns/default type nsfs (rw)\r\nshm on /var/lib/docker/containers/8f6668a380e8a13415ec38ded7eb663687cac3d137472c4675e87731f05a1d4e/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)\r\nmqueue on /var/lib/docker/containers/8f6668a380e8a13415ec38ded7eb663687cac3d137472c4675e87731f05a1d4e/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)\r\ntmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=94444k,mode=700)\r\n\r\n[root@rpi-slave01 ~]# docker --version\r\nDocker version 1.10.0, build 590d5108-unsupported\r\n\r\n[root@rpi-slave01 ~]# docker info\r\nContainers: 1\r\n Running: 1\r\n Paused: 0\r\n Stopped: 0\r\nImages: 12\r\nServer Version: 1.10.0\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nPlugins:\r\n Volume: local\r\n Network: null host bridge\r\nKernel Version: 4.4.10-1-ARCH\r\nOperating System: Arch Linux ARM\r\nOSType: linux\r\nArchitecture: armv7l\r\nCPUs: 4\r\nTotal Memory: 922.3 MiB\r\nName: rpi-slave01\r\nID: EQJY:RI3P:RY7Q:5SXB:KVAU:VTBA:FBDS:TGN5:4FDS:X45G:V2VZ:CZJN\r\nWARNING: No cpuset support\r\n\r\n[root@rpi-slave01 ~]# cat /proc/cgroups\r\n#subsys_name    hierarchy       num_cgroups     enabled\r\ncpuset  0       1       0\r\ncpu     3       11      1\r\ncpuacct 3       11      1\r\nblkio   6       9       1\r\nmemory  9       16      1\r\ndevices 4       51      1\r\nfreezer 2       5       1\r\nnet_cls 5       5       1\r\nperf_event      7       5       1\r\nnet_prio        5       5       1\r\npids    8       62      1\r\n\r\n[root@rpi-slave01 ~]# ll /sys/fs/cgroup/\r\ntotal 0\r\ndr-xr-xr-x 6 root root  0 Feb 22 15:40 blkio\r\nlrwxrwxrwx 1 root root 11 Feb 22 15:40 cpu -> cpu,cpuacct\r\ndr-xr-xr-x 6 root root  0 Feb 22 15:40 cpu,cpuacct\r\nlrwxrwxrwx 1 root root 11 Feb 22 15:40 cpuacct -> cpu,cpuacct\r\ndr-xr-xr-x 6 root root  0 Feb 22 15:40 devices\r\ndr-xr-xr-x 3 root root  0 Feb 22 15:40 freezer\r\ndr-xr-xr-x 6 root root  0 Feb 22 15:40 memory\r\nlrwxrwxrwx 1 root root 16 Feb 22 15:40 net_cls -> net_cls,net_prio\r\ndr-xr-xr-x 3 root root  0 Feb 22 15:40 net_cls,net_prio\r\nlrwxrwxrwx 1 root root 16 Feb 22 15:40 net_prio -> net_cls,net_prio\r\ndr-xr-xr-x 3 root root  0 Feb 22 15:40 perf_event\r\ndr-xr-xr-x 5 root root  0 Feb 22 15:40 pids\r\ndr-xr-xr-x 5 root root  0 Feb 22 15:40 systemd\r\n```\r\n\r\nStart of docker logs \r\n```\r\nW0515 14:21:55.904781    1242 server.go:445] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.\r\nW0515 14:21:55.905680    1242 server.go:406] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.\r\nI0515 14:21:55.906139    1242 plugins.go:71] No cloud provider specified.\r\nI0515 14:21:55.906197    1242 server.go:312] Successfully initialized cloud provider: \"\" from the config file: \"\"\r\nI0515 14:21:55.908295    1242 manager.go:132] cAdvisor running in container: \"/docker/8f6668a380e8a13415ec38ded7eb663687cac3d137472c4675e87731f05a1d4e\"\r\nI0515 14:21:55.952126    1242 fs.go:109] Filesystem partitions: map[/dev/root:{mountpoint:/var/lib/kubelet major:179 minor:2 fsType: blockSize:0}]\r\nE0515 14:21:55.956793    1242 machine.go:176] failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory\r\nI0515 14:21:56.716623    1242 manager.go:169] Machine: {NumCores:4 CpuFrequency:1200000 MemoryCapacity:967073792 MachineID:f21c916157f34d369cfa9e81943edba7 SystemUUID:f21c916157f34d369cfa9e81943edba7 BootID:9aac2194-927c-480c-a8ea-60dc6ed5bb3f Filesystems:[{Device:/dev/root Capacity:15513600000 Type:vfs Inodes:971040}] DiskMap:map[9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none} 179:0:{Name:mmcblk0 Major:179 Minor:0 Size:16003366912 Scheduler:noop}]\r\nNetworkDevices:[{Name:eth0 MacAddress:b8:27:eb:eb:41:90 Speed:10 Mtu:1500} {Name:wlan0 MacAddress:b8:27:eb:be:14:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:0 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]} {Id:2 Threads:[2] Caches:[]} {Id:3 Threads:[3] Caches:[]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}\r\nI0515 14:21:56.722124    1242 manager.go:175] Version: {KernelVersion:4.4.10-1-ARCH ContainerOsVersion:Raspbian GNU/Linux 8 (jessie) DockerVersion:1.10.0 CadvisorVersion: CadvisorRevision:}\r\nI0515 14:21:56.728270    1242 server.go:319] Using root directory: /var/lib/kubelet\r\nI0515 14:21:56.736232    1242 server.go:673] Adding manifest file: /etc/kubernetes/manifests-multi\r\nI0515 14:21:56.736509    1242 file.go:47] Watching path \"/etc/kubernetes/manifests-multi\"\r\nI0515 14:21:56.736632    1242 server.go:683] Watching apiserver\r\nW0515 14:21:56.865093    1242 kubelet.go:508] Hairpin mode set to \"promiscuous-bridge\" but configureCBR0 is false, falling back to \"hairpin-veth\"\r\nI0515 14:21:56.865294    1242 kubelet.go:276] Hairpin mode set to \"hairpin-veth\"\r\nI0515 14:21:56.917212    1242 manager.go:244] Setting dockerRoot to /var/lib/docker\r\nI0515 14:22:06.959767    1242 plugins.go:56] Registering credential provider: .dockercfg\r\nI0515 14:22:11.981754    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/aws-ebs\"\r\nI0515 14:22:11.982019    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/empty-dir\"\r\nI0515 14:22:11.982200    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/gce-pd\"\r\nI0515 14:22:11.982457    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/git-repo\"\r\nI0515 14:22:11.982641    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/host-path\"\r\nI0515 14:22:11.982801    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/nfs\"\r\nI0515 14:22:11.983029    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/secret\"\r\nI0515 14:22:11.983242    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/iscsi\"\r\nI0515 14:22:11.983452    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/glusterfs\"\r\nI0515 14:22:11.983659    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/persistent-claim\"\r\nI0515 14:22:11.983821    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/rbd\"\r\nI0515 14:22:11.984010    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/cinder\"\r\nI0515 14:22:11.984181    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/cephfs\"\r\nI0515 14:22:11.984395    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/downward-api\"\r\nI0515 14:22:11.984569    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/fc\"\r\nI0515 14:22:11.984806    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/flocker\"\r\nI0515 14:22:11.985001    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/azure-file\"\r\nI0515 14:22:11.985180    1242 plugins.go:291] Loaded volume plugin \"kubernetes.io/configmap\"\r\nI0515 14:22:11.985580    1242 server.go:632] Started kubelet\r\nE0515 14:22:11.986377    1242 kubelet.go:956] Image garbage collection failed: unable to find data for container /\r\nI0515 14:22:11.986625    1242 server.go:109] Starting to listen on 0.0.0.0:10250\r\nI0515 14:22:11.992270    1242 server.go:126] Starting to listen read-only on 0.0.0.0:10255\r\nE0515 14:22:12.009745    1242 event.go:202] Unable to write event: 'Post http://localhost:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)\r\nE0515 14:22:12.013200    1242 kubelet.go:1016] Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset]\r\nI0515 14:22:12.013374    1242 manager.go:123] Starting to sync pod status with apiserver\r\nI0515 14:22:12.013487    1242 kubelet.go:2356] Starting kubelet main sync loop.\r\nI0515 14:22:12.013573    1242 kubelet.go:2365] skipping pod synchronization - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset] container runtime is down]\r\nI0515 14:22:12.219074    1242 factory.go:230] Registering Docker factory\r\nI0515 14:22:12.230926    1242 factory.go:97] Registering Raw factory\r\nI0515 14:22:12.280502    1242 manager.go:1003] Started watching for new ooms in manager\r\nI0515 14:22:12.288396    1242 oomparser.go:182] oomparser using systemd\r\nI0515 14:22:12.291399    1242 manager.go:256] Starting recovery of all containers\r\nI0515 14:22:12.386913    1242 manager.go:261] Recovery completed\r\nI0515 14:22:17.013847    1242 kubelet.go:2365] skipping pod synchronization - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset]]\r\nE0515 14:22:19.352403    1242 event.go:202] Unable to write event: 'Post http://localhost:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)\r\nI0515 14:22:22.014463    1242 kubelet.go:2365] skipping pod synchronization - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset]]\r\nI0515 14:22:27.014808    1242 kubelet.go:2365] skipping pod synchronization - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset]]\r\nE0515 14:22:29.355421    1242 event.go:202] Unable to write event: 'Post http://localhost:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)\r\nI0515 14:22:32.015174    1242 kubelet.go:2365] skipping pod synchronization - [Failed to start ContainerManager system validation failed - Following Cgroup subsystem not mounted: [cpuset]]\r\n```"}}