{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin/issues/8190","repository_url":"https://api.github.com/repos/openshift/origin","labels_url":"https://api.github.com/repos/openshift/origin/issues/8190/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin/issues/8190/comments","events_url":"https://api.github.com/repos/openshift/origin/issues/8190/events","html_url":"https://github.com/openshift/origin/issues/8190","id":142632726,"number":8190,"title":"Nodes don't automatically rejoin cluster","user":{"login":"jeremyeder","id":1919640,"avatar_url":"https://avatars.githubusercontent.com/u/1919640?v=3","gravatar_id":"","url":"https://api.github.com/users/jeremyeder","html_url":"https://github.com/jeremyeder","followers_url":"https://api.github.com/users/jeremyeder/followers","following_url":"https://api.github.com/users/jeremyeder/following{/other_user}","gists_url":"https://api.github.com/users/jeremyeder/gists{/gist_id}","starred_url":"https://api.github.com/users/jeremyeder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeremyeder/subscriptions","organizations_url":"https://api.github.com/users/jeremyeder/orgs","repos_url":"https://api.github.com/users/jeremyeder/repos","events_url":"https://api.github.com/users/jeremyeder/events{/privacy}","received_events_url":"https://api.github.com/users/jeremyeder/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openshift/origin/labels/priority/P1","name":"priority/P1","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"pweil-","id":5230419,"avatar_url":"https://avatars.githubusercontent.com/u/5230419?v=3","gravatar_id":"","url":"https://api.github.com/users/pweil-","html_url":"https://github.com/pweil-","followers_url":"https://api.github.com/users/pweil-/followers","following_url":"https://api.github.com/users/pweil-/following{/other_user}","gists_url":"https://api.github.com/users/pweil-/gists{/gist_id}","starred_url":"https://api.github.com/users/pweil-/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pweil-/subscriptions","organizations_url":"https://api.github.com/users/pweil-/orgs","repos_url":"https://api.github.com/users/pweil-/repos","events_url":"https://api.github.com/users/pweil-/events{/privacy}","received_events_url":"https://api.github.com/users/pweil-/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2016-03-22T12:16:50Z","updated_at":"2016-03-23T19:33:25Z","closed_at":null,"body":"I rebooted a master and 2 nodes at roughly the same time.  The nodes happened to boot up faster (different hardware).  The master eventually came up (perhaps 1 minute after the nodes?).\r\n\r\nHowever the nodes are in NotReady status.  Looking at the logs for atomic-openshift-node service on one of the nodes, I see that the node tried to register with the master and couldn't reach it.\r\n\r\nDue to the Restart=always setting in the atomic-openshift-node unitfile, systemd restarted the service 5 times, and each could not connect.  Those 5 retries took a total of 5 seconds.  5 seconds is less than the reboot difference in this hardware, so the nodes did not rejoin.\r\n\r\nAfter the master is up, simply restarting atomic-openshift-node allows them to join the cluster and become Ready.\r\n\r\nI realize that there should be\r\n- multiple masters\r\n- sequenced/controlled cluster restart, reboot the master, wait for it to come back, then evac nodes and reboot them.\r\n\r\nBut, I think the systemd Restart=always is not enough to heal the cluster after such a situation where the single master goes down or reboots.\r\n\r\nI wonder if when the master comes up, it should go out and notify all nodes that it's back and signal them to rejoin? (maybe only makes sense in a single master environment)."},"comment":{"url":"https://api.github.com/repos/openshift/origin/issues/comments/200509452","html_url":"https://github.com/openshift/origin/issues/8190#issuecomment-200509452","issue_url":"https://api.github.com/repos/openshift/origin/issues/8190","id":200509452,"user":{"login":"pweil-","id":5230419,"avatar_url":"https://avatars.githubusercontent.com/u/5230419?v=3","gravatar_id":"","url":"https://api.github.com/users/pweil-","html_url":"https://github.com/pweil-","followers_url":"https://api.github.com/users/pweil-/followers","following_url":"https://api.github.com/users/pweil-/following{/other_user}","gists_url":"https://api.github.com/users/pweil-/gists{/gist_id}","starred_url":"https://api.github.com/users/pweil-/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pweil-/subscriptions","organizations_url":"https://api.github.com/users/pweil-/orgs","repos_url":"https://api.github.com/users/pweil-/repos","events_url":"https://api.github.com/users/pweil-/events{/privacy}","received_events_url":"https://api.github.com/users/pweil-/received_events","type":"User","site_admin":false},"created_at":"2016-03-23T19:33:25Z","updated_at":"2016-03-23T19:33:25Z","body":"@jeremyeder I'm curious as to why your systemd unit failed just because it could not connect to the master.  The kubelet should be able to handle an unreachable master without failing.  To verify I used the multi-node vagrant environment and stopped the `openshift-master` service while monitoring the `openshift-node` service on minion-1.  It didn't die, but failed to update node status as expected.  \r\n\r\nI also stopped the `openshift-node` service, waited for the master to show a not ready state and then restarted it to ensure it updated correctly to ready.  I think the issue here is that your service dies for some reason.  Any more info you can provide around that?\r\n\r\n@jawnsy node status syncs from the kubelet will retry indefinitely.  The retry interval is configurable but should be defaulting to every 10 seconds IIRC.  For each node sync the kubelet will attempt to send the update with a retry count before declaring that particular attempt a failure. "}}