{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubeadm/issues/1712","repository_url":"https://api.github.com/repos/kubernetes/kubeadm","labels_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/1712/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/1712/comments","events_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/1712/events","html_url":"https://github.com/kubernetes/kubeadm/issues/1712","id":477920399,"node_id":"MDU6SXNzdWU0Nzc5MjAzOTk=","number":1712,"title":"kubeadm join control-plane node times out","user":{"login":"chrischdi","id":417870,"node_id":"MDQ6VXNlcjQxNzg3MA==","avatar_url":"https://avatars0.githubusercontent.com/u/417870?v=4","gravatar_id":"","url":"https://api.github.com/users/chrischdi","html_url":"https://github.com/chrischdi","followers_url":"https://api.github.com/users/chrischdi/followers","following_url":"https://api.github.com/users/chrischdi/following{/other_user}","gists_url":"https://api.github.com/users/chrischdi/gists{/gist_id}","starred_url":"https://api.github.com/users/chrischdi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrischdi/subscriptions","organizations_url":"https://api.github.com/users/chrischdi/orgs","repos_url":"https://api.github.com/users/chrischdi/repos","events_url":"https://api.github.com/users/chrischdi/events{/privacy}","received_events_url":"https://api.github.com/users/chrischdi/received_events","type":"User","site_admin":false},"labels":[{"id":1488852105,"node_id":"MDU6TGFiZWwxNDg4ODUyMTA1","url":"https://api.github.com/repos/kubernetes/kubeadm/labels/area/etcd","name":"area/etcd","color":"0052cc","default":false},{"id":486837088,"node_id":"MDU6TGFiZWw0ODY4MzcwODg=","url":"https://api.github.com/repos/kubernetes/kubeadm/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":488818423,"node_id":"MDU6TGFiZWw0ODg4MTg0MjM=","url":"https://api.github.com/repos/kubernetes/kubeadm/labels/priority/important-longterm","name":"priority/important-longterm","color":"eb6420","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubeadm/milestones/13","html_url":"https://github.com/kubernetes/kubeadm/milestone/13","labels_url":"https://api.github.com/repos/kubernetes/kubeadm/milestones/13/labels","id":4275256,"node_id":"MDk6TWlsZXN0b25lNDI3NTI1Ng==","number":13,"title":"v1.16","description":"Placeholder for now","creator":{"login":"timothysc","id":169553,"node_id":"MDQ6VXNlcjE2OTU1Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/169553?v=4","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"open_issues":35,"closed_issues":29,"state":"open","created_at":"2019-04-30T19:45:12Z","updated_at":"2019-08-07T14:33:05Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2019-08-07T13:11:56Z","updated_at":"2019-08-07T15:11:27Z","closed_at":null,"author_association":"NONE","body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n## What keywords did you search in kubeadm issues before filing this one?\r\netcd join timeout\r\nkubeadm join timeout\r\n\r\n## Is this a BUG REPORT or FEATURE REQUEST?\r\nBUG REPORT\r\n\r\n## Versions\r\n\r\n**kubeadm version** (use `kubeadm version`): kubeadm version: &version.Info{Major:\"1\", Minor:\"15\", GitVersion:\"v1.15.2\", GitCommit:\"f6278300bebbb750328ac16ee6dd3aa7d3549568\", GitTreeState:\"clean\", BuildDate:\"2019-08-05T09:20:51Z\", GoVersion:\"go1.12.5\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\n**Environment**:\r\n- **Kubernetes version** (use `kubectl version`): v1.15.2\r\n- **Cloud provider or hardware configuration**: Openstack\r\n- **OS** (e.g. from /etc/os-release): Container Linux by CoreOS 2135.5.0 (Rhyolite)\r\n- **Kernel** (e.g. `uname -a`): Linux os1pi019-kube-master01 4.19.50-coreos-r1 #1 SMP Mon Jul 1 19:07:03 -00 2019 x86_64 Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz GenuineIntel GNU/Linux\r\n\r\n- **Others**:\r\n\r\n\r\n## What happened?\r\n`kubeadm join` was invoced and failed.\r\nThe etcd container did start up 7 seconds after kubeadm timed out / did exit with failure.\r\nSee the following logs (this include kubeadm logs and timestamps for pod-manifest starts):\r\n\r\n```\r\n09:30:27 kubeadm service starts\r\n09:30:27 kubeadm[2025]: [preflight] Reading configuration from the cluster...                                                                                                                                                                                                                                                                                   \r\n09:30:27 kubeadm[2025]: [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'                                                                                                                                                                                                                            \r\n09:30:27 kubeadm[2025]: [control-plane] Using manifest folder \"/etc/kubernetes/manifests\"                                                                                                                                                                                                                                                                       \r\n09:30:27 kubeadm[2025]: [control-plane] Creating static Pod manifest for \"kube-apiserver\"                                                                                                                                                                                                                                                                       \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"audit-policy\" to \"kube-apiserver\"                                                                                                                                                                                                                                                          \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"policy-controller\" to \"kube-apiserver\"                                                                                                                                                                                                                                                     \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"audit-log\" to \"kube-apiserver\"                                                                                                                                                                                                                                                             \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"scheduler-policy\" to \"kube-scheduler\"                                                                                                                                                                                                                                                      \r\n09:30:27 kubeadm[2025]: [control-plane] Creating static Pod manifest for \"kube-controller-manager\"                                                                                                                                                                                                                                                              \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"audit-policy\" to \"kube-apiserver\"                                                                                                                                                                                                                                                          \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"policy-controller\" to \"kube-apiserver\"                                                                                                                                                                                                                                                     \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"audit-log\" to \"kube-apiserver\"                                                                                                                                                                                                                                                             \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"scheduler-policy\" to \"kube-scheduler\"                                                                                                                                                                                                                                                      \r\n09:30:27 kubeadm[2025]: [control-plane] Creating static Pod manifest for \"kube-scheduler\"                                                                                                                                                                                                                                                                       \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"audit-policy\" to \"kube-apiserver\"                                                                                                                                                                                                                                                          \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"policy-controller\" to \"kube-apiserver\"                                                                                                                                                                                                                                                     \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"audit-log\" to \"kube-apiserver\"                                                                                                                                                                                                                                                             \r\n09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount \"scheduler-policy\" to \"kube-scheduler\"                                                                                                                                                                                                                                                      \r\n09:30:27 kubeadm[2025]: [check-etcd] Checking that the etcd cluster is healthy                                                                                                                                                                                                                                                                                  \r\n09:30:27 kubeadm[2025]: [kubelet-start] Downloading configuration for the kubelet from the \"kubelet-config-1.15\" ConfigMap in the kube-system namespace                                                                                                                                                                                                         \r\n09:30:27 kubeadm[2025]: [kubelet-start] Writing kubelet configuration to file \"/var/lib/kubelet/config.yaml\"                                                                                                                                                                                                                                                    \r\n09:30:27 kubeadm[2025]: [kubelet-start] Writing kubelet environment file with flags to file \"/var/lib/kubelet/kubeadm-flags.env\"                                                                                                                                                                                                                                \r\n09:30:27 kubeadm[2025]: [kubelet-start] Activating the kubelet service                                                                                                                                                                                                                                                                                          \r\n09:30:27 kubeadm[2025]: [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...                                                                                                                                                                                                                                                                 \r\n09:30:29 kubeadm[2025]: [etcd] Announced new etcd member joining to the existing etcd cluster                                                                                                                                                                                                                                                                   \r\n09:30:29 kubeadm[2025]: [etcd] Wrote Static Pod manifest for a local etcd member to \"/etc/kubernetes/manifests/etcd.yaml\"                                                                                                                                                                                                                                       \r\n09:30:29 kubeadm[2025]: [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s                                                                                                                                                                                                                                                     \r\n09:30:38 etcd pause shim create\r\n09:30:30 kube-scheduler pause shim create\r\n09:30:30 kube-controller-manager pause shim create\r\n09:30:34 kube-scheduler shim create\r\n09:30:35 kube-scheduler first logs\r\n09:30:36 kube-apiserver pause shim create\r\n09:31:07 kubeadm[2025]: [kubelet-check] Initial timeout of 40s passed.                                                                                                                                                                                                                                                                                          \r\n09:31:25 kube-controller-manager shim create\r\n09:31:25 kube-controller-manager first logs\r\n09:31:43 kube-apiserver shim create\r\n09:31:44 kubeadm[2025]: error execution phase control-plane-join/etcd: error creating local etcd static pod manifest file: timeout waiting for etcd cluster to be available                                                                                                                                                                                     \r\n09:31:44 systemd[1]: kubeadm.service: Main process exited, code=exited, status=1/FAILURE                                                                                                                                                                                                                                                                        \r\n09:31:44 kube-apiserver first logs\r\n09:31:51 etcd shim create\r\n09:31:52.081609 etcd first logs\r\n```\r\n\r\nThe timeout we hit here is [this one](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/etcd/local.go#L146) which uses hardcoded values (8 times 5 seconds -> 40s)\r\n\r\n## What you expected to happen?\r\nThe etcd member get's joined to the existing control-plane node and kubeadm succeeds.\r\n\r\n## How to reproduce it (as minimally and precisely as possible)?\r\n\r\nHard to say.\r\nTry lots of `kubeadm joins` of control-plane nodes\r\n\r\n## Anything else we need to know?\r\n\r\nIn `kubeadm init` there is a similar looking parameter called [TimeoutForControlPlane](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1beta2/types.go#L133)  which defaults to 4 Minutes and is used [here](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go#L87) to wait for the API server.\r\n\r\nThis is similar to me because the problem described here and the code at the kubeadm init  phase waits for a specific pod, started by the kubelet via a pod manifest.\r\n\r\nI see three options: \r\n* increase the hardcoded values\r\n* use the same parameter as already used during init (`TimeoutForControlPlane`) which would result in no change to the kubeadm specs\r\n* add an additional parameter to the kubeadm spec"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubeadm/issues/comments/519140864","html_url":"https://github.com/kubernetes/kubeadm/issues/1712#issuecomment-519140864","issue_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/1712","id":519140864,"node_id":"MDEyOklzc3VlQ29tbWVudDUxOTE0MDg2NA==","user":{"login":"chrischdi","id":417870,"node_id":"MDQ6VXNlcjQxNzg3MA==","avatar_url":"https://avatars0.githubusercontent.com/u/417870?v=4","gravatar_id":"","url":"https://api.github.com/users/chrischdi","html_url":"https://github.com/chrischdi","followers_url":"https://api.github.com/users/chrischdi/followers","following_url":"https://api.github.com/users/chrischdi/following{/other_user}","gists_url":"https://api.github.com/users/chrischdi/gists{/gist_id}","starred_url":"https://api.github.com/users/chrischdi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrischdi/subscriptions","organizations_url":"https://api.github.com/users/chrischdi/orgs","repos_url":"https://api.github.com/users/chrischdi/repos","events_url":"https://api.github.com/users/chrischdi/events{/privacy}","received_events_url":"https://api.github.com/users/chrischdi/received_events","type":"User","site_admin":false},"created_at":"2019-08-07T15:11:27Z","updated_at":"2019-08-07T15:11:27Z","author_association":"NONE","body":"> @chrischdi Several questions\r\n> \r\n> * Do you have the docker logs detailing what happened?\r\n\r\nNot anymore but we are running some builds every night and I will catch the logs on the next occurrence.\r\n\r\n> * Did the etcd container start?\r\nYes the etcd container was running from docker perspective. The kubernetes cluster is already deleted, so I don't know its exact state. I will also try to get more information on the next occurrence.\r\n> * What version of docker are you running?\r\nWe've got the coreos built-in docker version which is `18.06.3`\r\n> * Does this happen consistently or intermittently?\r\nI currently cannot reproduce it reliably.\r\n\r\nI hope to have it occure again to get all the details and more information.\r\n"}}
{"actor":{"display_login":"chrischdi"}}