{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28519","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28519/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28519/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28519/events","html_url":"https://github.com/kubernetes/kubernetes/issues/28519","id":163977914,"number":28519,"title":"Node Controller doesn't recheck that a node has the correct AWS tags on retry","user":{"login":"thomaso-mirodin","id":681004,"avatar_url":"https://avatars.githubusercontent.com/u/681004?v=3","gravatar_id":"","url":"https://api.github.com/users/thomaso-mirodin","html_url":"https://github.com/thomaso-mirodin","followers_url":"https://api.github.com/users/thomaso-mirodin/followers","following_url":"https://api.github.com/users/thomaso-mirodin/following{/other_user}","gists_url":"https://api.github.com/users/thomaso-mirodin/gists{/gist_id}","starred_url":"https://api.github.com/users/thomaso-mirodin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomaso-mirodin/subscriptions","organizations_url":"https://api.github.com/users/thomaso-mirodin/orgs","repos_url":"https://api.github.com/users/thomaso-mirodin/repos","events_url":"https://api.github.com/users/thomaso-mirodin/events{/privacy}","received_events_url":"https://api.github.com/users/thomaso-mirodin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-06T01:01:14Z","updated_at":"2016-07-06T01:01:14Z","closed_at":null,"body":"Context: I'm currently working on getting minions working with AWS Spot Fleet. Instead of using the workaround discussed in #24472 I decided to setup a service that would regularly poll the AWS API, list the instances and tag them as required (see: https://github.com/thomaso-mirodin/spot-fleet-tagger for my basic impl).\r\n\r\nIt seems that if a newly registered kubelet fails the first cloud provider health check, the node controller will never actually rerun the cloud provider health check fully unless the minion's kubelet process is restarted (via init or reboot). It seems that the retry loop the kubelet is in will at least attempt some partial rejoin, but because the node controller never seems to rerun the cloud provider healthcheck it will fail indefinitely.\r\n\r\nRelevant logs from the controller manager:\r\n\r\n```\r\nI0705 23:29:40.016523       1 nodecontroller.go:415] NodeController observed a new Node: api.Node{TypeMeta:unversioned.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:api.ObjectMeta{Name:\"ip-10-20-11-49.us-west-2.compute.internal\", GenerateName:\"\", Namespace:\"\", SelfLink:\"/api/v1/nodes/ip-10-20-11-49.us-west-2.compute.internal\", UID:\"579ae109-4308-11e6-8089-023c36c152d3\", ResourceVersion:\"42783938\", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603358178, nsec:0, loc:(*time.Location)(0x3b1c5c0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{<REMOVED>}, Images:[]api.ContainerImage(nil)}}\r\nI0705 23:29:40.016715       1 nodecontroller.go:603] Recording Registered Node ip-10-20-11-49.us-west-2.compute.internal in NodeController event message for node ip-10-20-11-49.us-west-2.compute.internal\r\nW0705 23:29:40.016787       1 nodecontroller.go:670] Missing timestamp for Node ip-10-20-11-49.us-west-2.compute.internal. Assuming now as a timestamp.\r\nI0705 23:29:40.016803       1 event.go:211] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"ip-10-20-11-49.us-west-2.compute.internal\", UID:\"ip-10-20-11-49.us-west-2.compute.internal\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node ip-10-20-11-49.us-west-2.compute.internal event: Registered Node ip-10-20-11-49.us-west-2.compute.internal in NodeController\r\nI0705 23:29:40.480742       1 nodecontroller.go:504] Deleting node (no longer present in cloud provider): ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:40.480766       1 nodecontroller.go:603] Recording Deleting Node ip-10-20-11-49.us-west-2.compute.internal because it's not present according to cloud provider event message for node ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:40.480812       1 event.go:211] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"ip-10-20-11-49.us-west-2.compute.internal\", UID:\"ip-10-20-11-49.us-west-2.compute.internal\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'DeletingNode' Node ip-10-20-11-49.us-west-2.compute.internal event: Deleting Node ip-10-20-11-49.us-west-2.compute.internal because it's not present according to cloud provider\r\nI0705 23:29:45.518965       1 nodecontroller.go:430] NodeController observed a Node deletion: ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.518997       1 nodecontroller.go:603] Recording Removing Node ip-10-20-11-49.us-west-2.compute.internal from NodeController event message for node ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.519197       1 event.go:211] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"ip-10-20-11-49.us-west-2.compute.internal\", UID:\"ip-10-20-11-49.us-west-2.compute.internal\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RemovingNode' Node ip-10-20-11-49.us-west-2.compute.internal event: Removing Node ip-10-20-11-49.us-west-2.compute.internal from NodeController\r\nI0705 23:29:45.597424       1 nodecontroller.go:603] Recording Deleting all Pods from Node ip-10-20-11-49.us-west-2.compute.internal. event message for node ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.597512       1 nodecontroller.go:839] Starting deletion of pod kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.597795       1 event.go:211] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"ip-10-20-11-49.us-west-2.compute.internal\", UID:\"ip-10-20-11-49.us-west-2.compute.internal\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'DeletingAllPods' Node ip-10-20-11-49.us-west-2.compute.internal event: Deleting all Pods from Node ip-10-20-11-49.us-west-2.compute.internal.\r\nI0705 23:29:45.598241       1 event.go:211] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\", UID:\"5a78bb56-4308-11e6-8089-023c36c152d3\", APIVersion:\"v1\", ResourceVersion:\"42783969\", FieldPath:\"\"}): type: 'Normal' reason: 'NodeControllerEviction' Marking for deletion Pod kube-proxy-ip-10-20-11-49.us-west-2.compute.internal from Node ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.618804       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nE0705 23:29:45.620287       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\nI0705 23:29:45.692231       1 nodecontroller.go:275] All pods terminated on ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.692256       1 nodecontroller.go:603] Recording Terminated all Pods on Node ip-10-20-11-49.us-west-2.compute.internal. event message for node ip-10-20-11-49.us-west-2.compute.internal\r\nI0705 23:29:45.692316       1 event.go:211] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"ip-10-20-11-49.us-west-2.compute.internal\", UID:\"ip-10-20-11-49.us-west-2.compute.internal\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'TerminatedAllPods' Node ip-10-20-11-49.us-west-2.compute.internal event: Terminated all Pods on Node ip-10-20-11-49.us-west-2.compute.internal.\r\nE0705 23:30:33.847780       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\nI0705 23:30:33.851718       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nI0705 23:30:34.785403       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nE0705 23:30:34.787754       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\nI0705 23:31:03.859249       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nE0705 23:31:03.861216       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\n# instance now has the required tags\r\nE0705 23:31:33.869395       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\nI0705 23:31:33.869942       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nE0705 23:31:33.894830       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\nI0705 23:32:03.878586       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nI0705 23:32:03.891587       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded\r\nE0705 23:32:03.892869       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\nE0705 23:32:33.888816       1 nodecontroller.go:352] pods \"kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\" not found\r\n# repeats forever\r\n```\r\n\r\nSorry if this is expected behavior or if I'm misunderstanding the steps involved when a kubelet joins the minion pool but I looked at the docs and the node controller code and neither gives me the impression that retries would not actually attempt to rejoin the cluster.\r\n* https://github.com/kubernetes/kubernetes/blob/master/docs/design/aws_under_the_hood.md#tagging\r\n* https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go#L634\r\n\r\nThere are a few workarounds available to me (reboot via API, cron job to check nodes labels and restart the kubelet, etc) but they are all less than ideal :D\r\n\r\n---\r\nVersions:\r\n```\r\n# Worker Kubelet\r\ncore@ip-10-20-0-10 ~ $ /opt/bin/kubelet --version\r\nKubernetes v1.2.4\r\n```\r\n\r\n```\r\n# Controller Manager's Hyberkube\r\nroot@ip-10-20-0-10:/# /hyperkube --version\r\nKubernetes v1.2.4\r\n```\r\n\r\nBoth were compiled from `3eed1e3be6848b877ff80a93da3785d9034d0a4f`"}}
{"actor":{"display_login":"thomaso-mirodin"}}