{"action":"closed","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/13508","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/13508/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/13508/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/13508/events","html_url":"https://github.com/rancher/rancher/issues/13508","id":323305096,"node_id":"MDU6SXNzdWUzMjMzMDUwOTY=","number":13508,"title":"sometimes rancher-agent doesn't get launched on an EC2 instance with `aws` cloud provider and correct iam profile","user":{"login":"sangeethah","id":4266958,"node_id":"MDQ6VXNlcjQyNjY5NTg=","avatar_url":"https://avatars0.githubusercontent.com/u/4266958?v=4","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"id":145695744,"node_id":"MDU6TGFiZWwxNDU2OTU3NDQ=","url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929","default":false},{"id":343948025,"node_id":"MDU6TGFiZWwzNDM5NDgwMjU=","url":"https://api.github.com/repos/rancher/rancher/labels/priority/1","name":"priority/1","color":"009800","default":false},{"id":203009589,"node_id":"MDU6TGFiZWwyMDMwMDk1ODk=","url":"https://api.github.com/repos/rancher/rancher/labels/status/to-test","name":"status/to-test","color":"66aa66","default":false},{"id":1040762509,"node_id":"MDU6TGFiZWwxMDQwNzYyNTA5","url":"https://api.github.com/repos/rancher/rancher/labels/team/az","name":"team/az","color":"f9d0c4","default":false},{"id":659821934,"node_id":"MDU6TGFiZWw2NTk4MjE5MzQ=","url":"https://api.github.com/repos/rancher/rancher/labels/version/2.0","name":"version/2.0","color":"e99695","default":false}],"state":"closed","locked":false,"assignee":{"login":"sangeethah","id":4266958,"node_id":"MDQ6VXNlcjQyNjY5NTg=","avatar_url":"https://avatars0.githubusercontent.com/u/4266958?v=4","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"assignees":[{"login":"sangeethah","id":4266958,"node_id":"MDQ6VXNlcjQyNjY5NTg=","avatar_url":"https://avatars0.githubusercontent.com/u/4266958?v=4","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},{"login":"StrongMonkey","id":9203378,"node_id":"MDQ6VXNlcjkyMDMzNzg=","avatar_url":"https://avatars0.githubusercontent.com/u/9203378?v=4","gravatar_id":"","url":"https://api.github.com/users/StrongMonkey","html_url":"https://github.com/StrongMonkey","followers_url":"https://api.github.com/users/StrongMonkey/followers","following_url":"https://api.github.com/users/StrongMonkey/following{/other_user}","gists_url":"https://api.github.com/users/StrongMonkey/gists{/gist_id}","starred_url":"https://api.github.com/users/StrongMonkey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrongMonkey/subscriptions","organizations_url":"https://api.github.com/users/StrongMonkey/orgs","repos_url":"https://api.github.com/users/StrongMonkey/repos","events_url":"https://api.github.com/users/StrongMonkey/events{/privacy}","received_events_url":"https://api.github.com/users/StrongMonkey/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/121","html_url":"https://github.com/rancher/rancher/milestone/121","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/121/labels","id":3213084,"node_id":"MDk6TWlsZXN0b25lMzIxMzA4NA==","number":121,"title":"v2.1","description":"","creator":{"login":"deniseschannon","id":11410997,"node_id":"MDQ6VXNlcjExNDEwOTk3","avatar_url":"https://avatars1.githubusercontent.com/u/11410997?v=4","gravatar_id":"","url":"https://api.github.com/users/deniseschannon","html_url":"https://github.com/deniseschannon","followers_url":"https://api.github.com/users/deniseschannon/followers","following_url":"https://api.github.com/users/deniseschannon/following{/other_user}","gists_url":"https://api.github.com/users/deniseschannon/gists{/gist_id}","starred_url":"https://api.github.com/users/deniseschannon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deniseschannon/subscriptions","organizations_url":"https://api.github.com/users/deniseschannon/orgs","repos_url":"https://api.github.com/users/deniseschannon/repos","events_url":"https://api.github.com/users/deniseschannon/events{/privacy}","received_events_url":"https://api.github.com/users/deniseschannon/received_events","type":"User","site_admin":false},"open_issues":104,"closed_issues":132,"state":"open","created_at":"2018-03-23T18:54:40Z","updated_at":"2018-09-11T23:27:58Z","due_on":null,"closed_at":null},"comments":16,"created_at":"2018-05-15T17:05:05Z","updated_at":"2018-09-11T23:27:58Z","closed_at":"2018-09-11T23:27:58Z","author_association":"MEMBER","body":"**Rancher versions: 2.0 Build from master - May 14\r\n\r\n**Steps to Reproduce:**\r\n\r\nCreate a EC2 cluster with following node configuration - AWS cloud provide. ( EC2 template has \"iamInstanceProfile\" set).\r\n\r\n1 control node\r\n1 etcd node\r\n3 worker nodes\r\n\r\nCluster is stuck in provisioning state forever with error message - \"workerPlane] Failed to bring up Worker Plane: Failed to verify healthcheck: Failed to check https://localhost:10250/healthz for service [kubelet] on host [<ip>]: Get https://localhost:10250/healthz: can not build dialer to c-qlvhr:m-kztpq\"\r\n\r\n\r\n\r\nrancher-server logs:\r\n\r\n```2018/05/15 14:50:53 [INFO] Provisioning cluster [c-qlvhr]\r\n2018/05/15 14:50:53 [INFO] Creating cluster [c-qlvhr]\r\n2018/05/15 14:50:53 [ERROR] Cluster c-qlvhr previously failed to create\r\n2018/05/15 14:50:53 [INFO] cluster [c-qlvhr] provisioning: Building Kubernetes cluster\r\n2018/05/15 14:50:53 [INFO] cluster [c-qlvhr] provisioning: [dialer] Setup tunnel for host [18.222.72.173]\r\n2018/05/15 14:50:53 [INFO] cluster [c-qlvhr] provisioning: [dialer] Setup tunnel for host [<ip>]\r\n2018/05/15 14:50:53 [INFO] cluster [c-qlvhr] provisioning: [state] Found local kube config file, trying to get state from cluster\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [state] Fetching cluster state from Kubernetes\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [state] Successfully Fetched cluster state to Kubernetes ConfigMap: cluster-state\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [certificates] Getting Cluster certificates from Kubernetes\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [certificates] Successfully fetched Cluster certificates from Kubernetes\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [network] No hosts added existing cluster, skipping port check\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [reconcile] Reconciling cluster state\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [reconcile] Check etcd hosts to be deleted\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [reconcile] Check etcd hosts to be added\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [reconcile] Rebuilding and updating local kube config\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: Successfully Deployed local admin kubeconfig at [management-state/rke/rke-108017177/kube_config_cluster.yml]\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [reconcile] host [<ip>] is active master on the cluster\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [reconcile] Reconciled cluster state successfully\r\n2018/05/15 14:50:54 [INFO] cluster [c-qlvhr] provisioning: [certificates] Deploying kubernetes certificates to Cluster nodes\r\n2018/05/15 14:50:59 [INFO] cluster [c-qlvhr] provisioning: Successfully Deployed local admin kubeconfig at [management-state/rke/rke-108017177/kube_config_cluster.yml]\r\n2018/05/15 14:50:59 [INFO] cluster [c-qlvhr] provisioning: [certificates] Successfully deployed kubernetes certificates to Cluster nodes\r\n2018/05/15 14:50:59 [INFO] cluster [c-qlvhr] provisioning: [cloud] Deploying cloud config file to node [18.222.72.173]\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [cloud] Successfully started [cloud-config-deployer] container on host [<ip>]\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [remove/cloud-config-deployer] Successfully removed container on host [<ip>]\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [cloud] Deploying cloud config file to node [18.218.89.6]\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [cloud] Successfully started [cloud-config-deployer] container on host [18.218.89.6]\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [remove/cloud-config-deployer] Successfully removed container on host [18.218.89.6]\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [cloud] Successfully deployed kubernetes cloud config to Cluster nodes\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: Pre-pulling kubernetes images\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: Kubernetes images pulled successfully\r\n2018/05/15 14:51:00 [INFO] cluster [c-qlvhr] provisioning: [etcd] Building up etcd plane..\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [etcd] Successfully started [rke-log-linker] container on host [18.222.72.173]\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [18.222.72.173]\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [etcd] Successfully started etcd plane..\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [controlplane] Building up Controller Plane..\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [sidekick] Sidekick container already created on host [18.218.89.6]\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.218.89.6]\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] service [kube-apiserver] on host [18.218.89.6] is healthy\r\n2018/05/15 14:51:01 [INFO] cluster [c-qlvhr] provisioning: [controlplane] Successfully started [rke-log-linker] container on host [18.218.89.6]\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [18.218.89.6]\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.218.89.6]\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] service [kube-controller-manager] on host [18.218.89.6] is healthy\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [controlplane] Successfully started [rke-log-linker] container on host [18.218.89.6]\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [18.218.89.6]\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.218.89.6]\r\n2018/05/15 14:51:02 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] service [kube-scheduler] on host [18.218.89.6] is healthy\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [controlplane] Successfully started [rke-log-linker] container on host [18.218.89.6]\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [18.218.89.6]\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [controlplane] Successfully started Controller Plane..\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [authz] Creating rke-job-deployer ServiceAccount\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [authz] rke-job-deployer ServiceAccount created successfully\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [authz] Creating system:node ClusterRoleBinding\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [authz] system:node ClusterRoleBinding created successfully\r\n2018/05/15 14:51:03 [INFO] cluster [c-qlvhr] provisioning: [certificates] Save kubernetes certificates as secrets\r\n2018/05/15 14:51:04 [INFO] cluster [c-qlvhr] provisioning: [certificates] Successfully saved certificates as kubernetes secret [k8s-certs]\r\n2018/05/15 14:51:04 [INFO] cluster [c-qlvhr] provisioning: [state] Saving cluster state to Kubernetes\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [worker] Building up Worker Plane..\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [sidekick] Sidekick container already created on host [18.218.89.6]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] Start Healthcheck on service [kubelet] on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] service [kubelet] on host [<ip>] is healthy\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [worker] Successfully started [rke-log-linker] container on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [worker] Successfully started [rke-log-linker] container on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [sidekick] Sidekick container already created on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] Start Healthcheck on service [kubelet] on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] Start Healthcheck on service [kube-proxy] on host [<ip>]\r\n2018/05/15 14:51:05 [INFO] cluster [c-qlvhr] provisioning: [healthcheck] service [kube-proxy] on host [<ip>] is healthy\r\n2018/05/15 14:51:06 [INFO] cluster [c-qlvhr] provisioning: [worker] Successfully started [rke-log-linker] container on host [18.218.89.6]\r\n2018/05/15 14:51:06 [INFO] cluster [c-qlvhr] provisioning: [remove/rke-log-linker] Successfully removed container on host [<ip>]\r\n2018/05/15 14:51:55 [ERROR] cluster [c-qlvhr] provisioning: [workerPlane] Failed to bring up Worker Plane: Failed to verify healthcheck: Failed to check https://localhost:10250/healthz for service [kubelet] on host [<ip>]: Get https://localhost:10250/healthz: can not build dialer to c-qlvhr:m-kztpq``"}}
{"actor":{"display_login":"sangeethah"}}