{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/issues/514","repository_url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere","labels_url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/issues/514/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/issues/514/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/issues/514/events","html_url":"https://github.com/kubernetes/kubernetes-anywhere/pull/514","id":291753509,"number":514,"title":"Kubeadm installs fail with removal of kubelet and kubeadm","user":{"login":"leblancd","id":3209677,"avatar_url":"https://avatars2.githubusercontent.com/u/3209677?v=4","gravatar_id":"","url":"https://api.github.com/users/leblancd","html_url":"https://github.com/leblancd","followers_url":"https://api.github.com/users/leblancd/followers","following_url":"https://api.github.com/users/leblancd/following{/other_user}","gists_url":"https://api.github.com/users/leblancd/gists{/gist_id}","starred_url":"https://api.github.com/users/leblancd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leblancd/subscriptions","organizations_url":"https://api.github.com/users/leblancd/orgs","repos_url":"https://api.github.com/users/leblancd/repos","events_url":"https://api.github.com/users/leblancd/events{/privacy}","received_events_url":"https://api.github.com/users/leblancd/received_events","type":"User","site_admin":false},"labels":[{"id":486150700,"url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":678511754,"url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/labels/size/XS","name":"size/XS","color":"009900","default":false}],"state":"open","locked":false,"assignee":{"login":"luxas","id":5442475,"avatar_url":"https://avatars1.githubusercontent.com/u/5442475?v=4","gravatar_id":"","url":"https://api.github.com/users/luxas","html_url":"https://github.com/luxas","followers_url":"https://api.github.com/users/luxas/followers","following_url":"https://api.github.com/users/luxas/following{/other_user}","gists_url":"https://api.github.com/users/luxas/gists{/gist_id}","starred_url":"https://api.github.com/users/luxas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luxas/subscriptions","organizations_url":"https://api.github.com/users/luxas/orgs","repos_url":"https://api.github.com/users/luxas/repos","events_url":"https://api.github.com/users/luxas/events{/privacy}","received_events_url":"https://api.github.com/users/luxas/received_events","type":"User","site_admin":false},"assignees":[{"login":"pipejakob","id":1605981,"avatar_url":"https://avatars2.githubusercontent.com/u/1605981?v=4","gravatar_id":"","url":"https://api.github.com/users/pipejakob","html_url":"https://github.com/pipejakob","followers_url":"https://api.github.com/users/pipejakob/followers","following_url":"https://api.github.com/users/pipejakob/following{/other_user}","gists_url":"https://api.github.com/users/pipejakob/gists{/gist_id}","starred_url":"https://api.github.com/users/pipejakob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pipejakob/subscriptions","organizations_url":"https://api.github.com/users/pipejakob/orgs","repos_url":"https://api.github.com/users/pipejakob/repos","events_url":"https://api.github.com/users/pipejakob/events{/privacy}","received_events_url":"https://api.github.com/users/pipejakob/received_events","type":"User","site_admin":false},{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars3.githubusercontent.com/u/1787169?v=4","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},{"login":"luxas","id":5442475,"avatar_url":"https://avatars1.githubusercontent.com/u/5442475?v=4","gravatar_id":"","url":"https://api.github.com/users/luxas","html_url":"https://github.com/luxas","followers_url":"https://api.github.com/users/luxas/followers","following_url":"https://api.github.com/users/luxas/following{/other_user}","gists_url":"https://api.github.com/users/luxas/gists{/gist_id}","starred_url":"https://api.github.com/users/luxas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luxas/subscriptions","organizations_url":"https://api.github.com/users/luxas/orgs","repos_url":"https://api.github.com/users/luxas/repos","events_url":"https://api.github.com/users/luxas/events{/privacy}","received_events_url":"https://api.github.com/users/luxas/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-01-25T23:34:03Z","updated_at":"2018-02-18T15:40:15Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/pulls/514","html_url":"https://github.com/kubernetes/kubernetes-anywhere/pull/514","diff_url":"https://github.com/kubernetes/kubernetes-anywhere/pull/514.diff","patch_url":"https://github.com/kubernetes/kubernetes-anywhere/pull/514.patch"},"body":"## Problem Description:\r\nKubeadm installations are failing on the master node. Looking at\r\nstartup logs on the master node:\r\n\r\n```\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script: dpkg: error processing archive /tmp/k8s-debs/kubernetes-cni.deb (--install):\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:  error creating directory '/opt/cni/bin': No such file or directory\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script: dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script: dpkg: dependency problems prevent configuration of kubelet:\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:  kubelet depends on kubernetes-cni (>= 0.5.1); however:\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:   Package kubernetes-cni is not installed.\r\n```\r\nfollowed by:\r\n```\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:  /tmp/k8s-debs/kubernetes-cni.deb\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:  kubelet\r\nJan 23 01:56:33 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:  kubeadm\r\n```\r\nand:\r\n```\r\nJan 23 01:56:35 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script: The following packages will be REMOVED:\r\nJan 23 01:56:35 e2e-9c7300d2ef-master startup-script[1785]: INFO startup-script:   kubeadm kubelet\r\n```\r\n\r\nApparently what's happening is this:\r\n- Startup script tries to install kubernetes-cni, kubelet, and kubeadm from debs packages\r\n- Installation of kubernetes-cni fails because the directory /opt/cni/bin can't be created\r\n- Installation of kubelet fails because it depends upon kubernetes-cni being installed\r\n- Installation of kubeadm fails because it depends upon kubelet being installed\r\n- The 'apt-get install -f -y' cmd detects these dependency failures and remove kubelet and kubeadm\r\n- Because kubeadm is missing, the 'kubeadm init ...' command can't be run, so that the kubeconfig in /etc/kubernetes is never created\r\n- Because the kubeconfig is never created, kubernetes-anywhere fails after 50 (or is it 60, I forget) attempts to read the kubeconfig.\r\n\r\n## Fix:\r\nPreemptively adding a creation of a directory /opt/cni/bin before installing\r\nkubernetes-cni fixes the problem.\r\n\r\n## Log with Fix Showing Success:\r\n[Successful Log Snippet](https://gist.github.com/leblancd/f596e4b34f4768efa7748d3cca4a062e)\r\n\r\nfixes #513 \r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/issues/comments/366524728","html_url":"https://github.com/kubernetes/kubernetes-anywhere/pull/514#issuecomment-366524728","issue_url":"https://api.github.com/repos/kubernetes/kubernetes-anywhere/issues/514","id":366524728,"user":{"login":"leblancd","id":3209677,"avatar_url":"https://avatars2.githubusercontent.com/u/3209677?v=4","gravatar_id":"","url":"https://api.github.com/users/leblancd","html_url":"https://github.com/leblancd","followers_url":"https://api.github.com/users/leblancd/followers","following_url":"https://api.github.com/users/leblancd/following{/other_user}","gists_url":"https://api.github.com/users/leblancd/gists{/gist_id}","starred_url":"https://api.github.com/users/leblancd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leblancd/subscriptions","organizations_url":"https://api.github.com/users/leblancd/orgs","repos_url":"https://api.github.com/users/leblancd/repos","events_url":"https://api.github.com/users/leblancd/events{/privacy}","received_events_url":"https://api.github.com/users/leblancd/received_events","type":"User","site_admin":false},"created_at":"2018-02-18T15:40:15Z","updated_at":"2018-02-18T15:40:15Z","author_association":"NONE","body":"@Lion-Wei - The '-f' flag in the 'apt-get install -f -y' command is a short form for '--fix-broken'. At the time this gets called, kubelet and kubeadm have been \"brokenly\" installed because they are missing dependencies (kubelet depends on kubernetes-cni, which is missing, and kubeadm depends upon kubelet, which has missing dependencies). I believe that what happens when 'apt-get install -f -y' is called is that apt-get does its best to fix the broken install of kubelet and kubeadm, and when it can't, it removes the packages (i.e. it's better to not have a broken package, than to leave it and end up with complex, possibly indeterminate behavior because of the missing dependencies)."}}
{"actor":{"display_login":"leblancd"}}