{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/1105","repository_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws","labels_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/1105/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/1105/comments","events_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/1105/events","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/1105","id":286451740,"number":1105,"title":"'kube-node-drainer' does not work 'kube2iam' in 0.9.9","user":{"login":"whereisaaron","id":1109585,"avatar_url":"https://avatars0.githubusercontent.com/u/1109585?v=4","gravatar_id":"","url":"https://api.github.com/users/whereisaaron","html_url":"https://github.com/whereisaaron","followers_url":"https://api.github.com/users/whereisaaron/followers","following_url":"https://api.github.com/users/whereisaaron/following{/other_user}","gists_url":"https://api.github.com/users/whereisaaron/gists{/gist_id}","starred_url":"https://api.github.com/users/whereisaaron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whereisaaron/subscriptions","organizations_url":"https://api.github.com/users/whereisaaron/orgs","repos_url":"https://api.github.com/users/whereisaaron/repos","events_url":"https://api.github.com/users/whereisaaron/events{/privacy}","received_events_url":"https://api.github.com/users/whereisaaron/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":40,"created_at":"2018-01-06T02:18:21Z","updated_at":"2018-02-08T12:15:55Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I have `kube-aws` 0.9.9 clusters (us-east-2) with `kube-node-drainer` enabled and running, but it doesn't actually appear to be working. I don't see any sign of pods draining during ASG termination. The controller doesn't reschedule any pods until well after the instance has terminated and the `nodeMonitorGracePeriod` has elapsed (which I set to 90s to test). All the pods then get rescheduled at the same instant.\r\n\r\nThe ASG instances spend 5-10 minutes in `Terminating: Pending` state before transitioning to `Terminating: Proceed`. At that point all connection to the instance drops. But only a couple minutes later do any pods get rescheduled.\r\n\r\nA log from a `kube-node-drainer` on a node being terminated is below. It gets a 404 response from the AWS API, right up to the point when the connection drops.\r\n\r\nThe clusters have `kube2iam` installed, does that interfer or block the AWS API access for `kube-node-drainer`?\r\n\r\nOutput of `kubectl -n kube-system logs kube-node-drainer-ds-289hr --follow`\r\n```\r\n<=== At this point the instance is \"Terminating: Wait\" in ASG console\r\n+ curl -o /dev/null -w %{http_code} -sL http://169.254.169.254/latest/meta-data/spot/termination-time\r\n+ http_status=404\r\n+ [ 404 -eq 200 ]\r\n+ [ -e /etc/kube-node-drainer/asg ]\r\n+ grep -q i-08fc5fb751b30d164 /etc/kube-node-drainer/asg\r\n+ sleep 10\r\n+ curl -o /dev/null -w %{http_code} -sL http://169.254.169.254/latest/meta-data/spot/termination-time\r\n+ http_status=404\r\n+ [ 404 -eq 200 ]\r\n+ [ -e /etc/kube-node-drainer/asg ]\r\n+ grep -q i-08fc5fb751b30d164 /etc/kube-node-drainer/asg\r\n+ sleep 10\r\n+ curl -o /dev/null -w %{http_code} -sL http://169.254.169.254/latest/meta-data/spot/termination-time\r\n+ http_status=404\r\n+ [ 404 -eq 200 ]\r\n+ [ -e /etc/kube-node-drainer/asg ]\r\n+ grep -q i-08fc5fb751b30d164 /etc/kube-node-drainer/asg\r\n+ sleep 10\r\n+ curl -o /dev/null -w %{http_code} -sL http://169.254.169.254/latest/meta-data/spot/termination-time\r\n+ http_status=404\r\n+ [ 404 -eq 200 ]\r\n+ [ -e /etc/kube-node-drainer/asg ]\r\n+ grep -q i-08fc5fb751b30d164 /etc/kube-node-drainer/asg\r\n+ sleep 10\r\n+ curl -o /dev/null -w %{http_code} -sL http://169.254.169.254/latest/meta-data/spot/termination-time\r\n+ http_status=404\r\n+ [ 404 -eq 200 ]\r\n+ [ -e /etc/kube-node-drainer/asg ]\r\n+ grep -q i-08fc5fb751b30d164 /etc/kube-node-drainer/asg\r\n+ sleep 10\r\n+ curl -o /dev/null -w %{http_code} -sL http://169.254.169.254/latest/meta-data/spot/termination-time\r\n+ http_status=404\r\n+ [ 404 -eq 200 ]\r\n+ [ -e /etc/kube-node-drainer/asg ]\r\n+ grep -q i-08fc5fb751b30d164 /etc/kube-node-drainer/asg\r\n+ sleep 10\r\n<=== At this point the connection is cut and instance is \"Terminating: Proceed\" in ASG console\r\n[a minute or so passes]\r\n<=== At this point the instance is terminated, but still showed as 'Ready' in `kubectl get nodes`\r\n[a minute later]\r\n<=== Instance drops from `kubectl get nodes` and all the pods are rescheduled\r\n```\r\n\r\n  "},"comment":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/comments/364094414","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/1105#issuecomment-364094414","issue_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/1105","id":364094414,"user":{"login":"c-knowles","id":3053013,"avatar_url":"https://avatars3.githubusercontent.com/u/3053013?v=4","gravatar_id":"","url":"https://api.github.com/users/c-knowles","html_url":"https://github.com/c-knowles","followers_url":"https://api.github.com/users/c-knowles/followers","following_url":"https://api.github.com/users/c-knowles/following{/other_user}","gists_url":"https://api.github.com/users/c-knowles/gists{/gist_id}","starred_url":"https://api.github.com/users/c-knowles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c-knowles/subscriptions","organizations_url":"https://api.github.com/users/c-knowles/orgs","repos_url":"https://api.github.com/users/c-knowles/repos","events_url":"https://api.github.com/users/c-knowles/events{/privacy}","received_events_url":"https://api.github.com/users/c-knowles/received_events","type":"User","site_admin":false},"created_at":"2018-02-08T12:15:55Z","updated_at":"2018-02-08T12:15:55Z","author_association":"COLLABORATOR","body":"@pingles thanks for adding so much detail here! Between us it seems we'll be able to work this out pretty soon.\r\n\r\n> @c-knowles it's not something I was aware of. Do you mean there's a situation where that rule removal doesn't apply/work, or that it's not sufficient for all cases?\r\n\r\nTo clarify that, I just meant that the `defer` would only be actioned during normal kiam app shutdown and that the issues described above that by the others would probably not be solved by that defer code."}}
{"actor":{"display_login":"c-knowles"}}