{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kops/issues/6154","repository_url":"https://api.github.com/repos/kubernetes/kops","labels_url":"https://api.github.com/repos/kubernetes/kops/issues/6154/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kops/issues/6154/comments","events_url":"https://api.github.com/repos/kubernetes/kops/issues/6154/events","html_url":"https://github.com/kubernetes/kops/issues/6154","id":387264423,"node_id":"MDU6SXNzdWUzODcyNjQ0MjM=","number":6154,"title":"AWS authenticator stops master from joining cluster","user":{"login":"phillipj","id":1231635,"node_id":"MDQ6VXNlcjEyMzE2MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/1231635?v=4","gravatar_id":"","url":"https://api.github.com/users/phillipj","html_url":"https://github.com/phillipj","followers_url":"https://api.github.com/users/phillipj/followers","following_url":"https://api.github.com/users/phillipj/following{/other_user}","gists_url":"https://api.github.com/users/phillipj/gists{/gist_id}","starred_url":"https://api.github.com/users/phillipj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/phillipj/subscriptions","organizations_url":"https://api.github.com/users/phillipj/orgs","repos_url":"https://api.github.com/users/phillipj/repos","events_url":"https://api.github.com/users/phillipj/events{/privacy}","received_events_url":"https://api.github.com/users/phillipj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-12-04T12:32:33Z","updated_at":"2018-12-04T21:35:55Z","closed_at":null,"author_association":"NONE","body":"**1. What `kops` version are you running? The command `kops version`, will display\r\n this information.**\r\n\r\n```\r\n$ kops version\r\nVersion 1.10.0\r\n```\r\n\r\n**2. What Kubernetes version are you running? `kubectl version` will print the\r\n version if a cluster is running or provide the Kubernetes version specified as\r\n a `kops` flag.**\r\n\r\n```\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"12\", GitVersion:\"v1.12.2\", GitCommit:\"17c77c7898218073f14c8d573582e8d2313dc740\", GitTreeState:\"clean\", BuildDate:\"2018-10-30T21:39:38Z\", GoVersion:\"go1.11.1\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"11\", GitVersion:\"v1.11.3\", GitCommit:\"a4529464e4629c21224b3d52edfe0ea91b072862\", GitTreeState:\"clean\", BuildDate:\"2018-09-09T17:53:03Z\", GoVersion:\"go1.10.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n**3. What cloud provider are you using?**\r\n\r\nAWS\r\n\r\n**4. What commands did you run?  What is the simplest way to reproduce this issue?**\r\n\r\n```\r\n$ kops edit cluster\r\n# .. added authentication section\r\n$ kops rolling-update cluster --instance-group-roles=Master --force --yes\r\n```\r\n\r\nThe authentication section added:\r\n\r\n```yaml\r\nauthentication:\r\n  aws: {}\r\n```\r\nas described in: [./docs/authentication.md#aws-iam-authenticator](https://github.com/kubernetes/kops/blob/master/docs/authentication.md#aws-iam-authenticator)\r\n\r\n**5. What happened after the commands executed?**\r\n\r\n`kops` drained and stopped the first master node. A new EC2 instance was created by AWS auto scaling group, but that instance was never able to join the cluster. The cluster validation step therefore failed.\r\n\r\n```\r\n$ kops rolling-update cluster --instance-group-roles=Master --force --yes\r\n\r\nI1204 11:13:18.976387   85319 instancegroups.go:157] Draining the node: \"ip-172-31-26-101.x.y.z\".\r\nnode \"ip-172-31-26-101.x.y.z\" cordoned\r\nnode \"ip-172-31-26-101.x.y.z\" cordoned\r\nnode \"ip-172-31-26-101.x.y.z\" drained\r\nI1204 11:13:34.924664   85319 instancegroups.go:338] Waiting for 1m30s for pods to stabilize after draining.\r\nI1204 11:15:04.911031   85319 instancegroups.go:278] Stopping instance \"i-0d7d0c586f9a7\", node \"ip-172-31-26-101.x.y.z\", in group \"master-x.masters.k8s.y.z\" (this may take a while).\r\nI1204 11:20:06.535982   85319 instancegroups.go:188] Validating the cluster.\r\nI1204 11:20:14.653181   85319 instancegroups.go:251] Cluster did not pass validation, will try again in \"30s\" until duration \"5m0s\" expires: machine \"i-04ace14bd333d8\" has not yet joined cluster.\r\nI1204 11:20:48.953718   85319 instancegroups.go:251] Cluster did not pass validation, will try again in \"30s\" until duration \"5m0s\" expires: machine \"i-04ace14bd333d8\" has not yet joined cluster.\r\nI1204 11:21:18.639329   85319 instancegroups.go:251] Cluster did not pass validation, will try again in \"30s\" until duration \"5m0s\" expires: machine \"i-04ace14bd333d8\" has not yet joined cluster.\r\nI1204 11:21:48.260554   85319 instancegroups.go:251] Cluster did not pass validation, will try again in \"30s\" until duration \"5m0s\" expires: machine \"i-04ace14bd333d8\" has not yet joined cluster.\r\nI1204 11:22:18.447799   85319 instancegroups.go:251] Cluster did not pass validation, will try again in \"30s\" until duration \"5m0s\" expires: machine \"i-04ace14bd333d8\" has not yet joined cluster.\r\n\r\nmaster not healthy after update, stopping rolling-update: \"error validating cluster after removing a node: cluster did not validate within a duation of \\\"5m0s\\\"\"\r\n```\r\n\r\nSeeing this in the master nodes logs:\r\n\r\n```\r\nmain.go:142] got error running nodeup (will retry in 30s): error building loader: certificate \"aws-iam-authenticator\" not found\r\ns3fs.go:219] Reading file \"s3://x-kops-state/k8s.x.y.z/pki/issued/aws-iam-authenticator/keyset.yaml\"\r\n```\r\n\r\n**6. What did you expect to happen?**\r\n\r\nAll the master nodes to be updated with the `aws-iam-authenticator` enabled & ready for action.\r\n\r\n/cc @rdrgmnzs"},"comment":{"url":"https://api.github.com/repos/kubernetes/kops/issues/comments/444267596","html_url":"https://github.com/kubernetes/kops/issues/6154#issuecomment-444267596","issue_url":"https://api.github.com/repos/kubernetes/kops/issues/6154","id":444267596,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDI2NzU5Ng==","user":{"login":"qlikcoe","id":33262478,"node_id":"MDQ6VXNlcjMzMjYyNDc4","avatar_url":"https://avatars1.githubusercontent.com/u/33262478?v=4","gravatar_id":"","url":"https://api.github.com/users/qlikcoe","html_url":"https://github.com/qlikcoe","followers_url":"https://api.github.com/users/qlikcoe/followers","following_url":"https://api.github.com/users/qlikcoe/following{/other_user}","gists_url":"https://api.github.com/users/qlikcoe/gists{/gist_id}","starred_url":"https://api.github.com/users/qlikcoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qlikcoe/subscriptions","organizations_url":"https://api.github.com/users/qlikcoe/orgs","repos_url":"https://api.github.com/users/qlikcoe/repos","events_url":"https://api.github.com/users/qlikcoe/events{/privacy}","received_events_url":"https://api.github.com/users/qlikcoe/received_events","type":"User","site_admin":false},"created_at":"2018-12-04T21:35:55Z","updated_at":"2018-12-04T21:35:55Z","author_association":"NONE","body":"Also, kops doc also doesn't say you have to do anything else other than adding `aws: {}` piece:\r\nhttps://github.com/kubernetes/kops/blob/master/docs/authentication.md"}}
{"actor":{"display_login":"qlikcoe"}}