{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/12809","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/12809/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/12809/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/12809/events","html_url":"https://github.com/rancher/rancher/issues/12809","id":315286729,"number":12809,"title":"cluster create on EC2 ends with \"Waiting to register with Kubernetes\"","user":{"login":"mmorianos","id":4206393,"avatar_url":"https://avatars2.githubusercontent.com/u/4206393?v=4","gravatar_id":"","url":"https://api.github.com/users/mmorianos","html_url":"https://github.com/mmorianos","followers_url":"https://api.github.com/users/mmorianos/followers","following_url":"https://api.github.com/users/mmorianos/following{/other_user}","gists_url":"https://api.github.com/users/mmorianos/gists{/gist_id}","starred_url":"https://api.github.com/users/mmorianos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mmorianos/subscriptions","organizations_url":"https://api.github.com/users/mmorianos/orgs","repos_url":"https://api.github.com/users/mmorianos/repos","events_url":"https://api.github.com/users/mmorianos/events{/privacy}","received_events_url":"https://api.github.com/users/mmorianos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-18T00:26:10Z","updated_at":"2018-04-18T01:10:19Z","closed_at":null,"author_association":"NONE","body":"**Rancher versions:**\r\nrancher/server: 2.0.0-beta3 on 16.04.1 LTS\r\n\r\nTrying vanilla deployment into AWS on 16.04.1 LTS node.  Tried multiple node counts with various role assignments before pulling back to single node deployment, with all roles.\r\n\r\n![waiting](https://user-images.githubusercontent.com/4206393/38905203-1e6f24ae-427d-11e8-86c8-d19b4bb9bbb9.jpeg)\r\n\r\nrancher/server logs below...\r\n\r\nI0417 23:50:08.873521       1 http.go:105] HTTP2 has been explicitly disabled\r\n2018/04/17 23:50:08 [INFO] [certificates] Generating CA kubernetes certificates\r\n2018/04/17 23:50:09 [INFO] [certificates] Generating Kubernetes API server certificates\r\n2018/04/17 23:50:09 [INFO] [certificates] Generating Kube Controller certificates\r\n2018/04/17 23:50:09 [INFO] [certificates] Generating Kube Scheduler certificates\r\n2018/04/17 23:50:10 [INFO] [certificates] Generating Kube Proxy certificates\r\n2018/04/17 23:50:10 [INFO] [certificates] Generating Node certificate\r\n2018/04/17 23:50:10 [INFO] [certificates] Generating admin certificates and kubeconfig\r\n2018/04/17 23:50:10 [INFO] [certificates] Generating etcd-127.0.0.1 certificate and key\r\n2018/04/17 23:50:11 [INFO] Running kube-apiserver --bind-address=127.0.0.1 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --advertise-address=10.43.0.1 --insecure-bind-address=127.0.0.1 --admission-control=ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds --storage-backend=etcd3 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --insecure-port=0 --allow-privileged=true --service-cluster-ip-range=10.43.0.0/16 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --authorization-mode=Node,RBAC --secure-port=6443 --cloud-provider= --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-servers=https://127.0.0.1:2379 --etcd-prefix=/registry -v=1 --logtostderr=false --alsologtostderr=false\r\n2018/04/17 23:50:11 [INFO] Activating driver gke\r\n2018/04/17 23:50:11 [INFO] Activating driver gke done\r\n2018/04/17 23:50:11 [INFO] Activating driver rke\r\n2018/04/17 23:50:11 [INFO] Activating driver rke done\r\n2018/04/17 23:50:11 [INFO] Activating driver aks\r\n2018/04/17 23:50:11 [INFO] Activating driver aks done\r\n2018/04/17 23:50:11 [INFO] Activating driver import\r\n2018/04/17 23:50:11 [INFO] Activating driver import done\r\n2018/04/17 23:50:11 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r\n2018/04/17 23:50:11 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r\n2018/04/17 23:50:11 [INFO] Running etcd --peer-client-cert-auth --client-cert-auth --listen-client-urls=https://0.0.0.0:2379 --data-dir=/var/lib/rancher/etcd --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem --name=etcd-master --initial-advertise-peer-urls=https://127.0.0.1:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem --listen-peer-urls=https://0.0.0.0:2380 --initial-cluster-token=etcd-cluster-1 --advertise-client-urls=https://127.0.0.1:2379,https://127.0.0.1:4001 --initial-cluster=etcd-master=https://127.0.0.1:2380 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem\r\n2018-04-17 23:50:11.429016 I | etcdmain: etcd Version: 3.2.13\r\n2018-04-17 23:50:11.429122 I | etcdmain: Git SHA: Not provided (use ./build instead of go build)\r\n2018-04-17 23:50:11.429188 I | etcdmain: Go Version: go1.9.2\r\n2018-04-17 23:50:11.429275 I | etcdmain: Go OS/Arch: linux/amd64\r\n2018-04-17 23:50:11.429355 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2\r\n2018-04-17 23:50:11.429624 I | embed: peerTLS: cert = /etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem, key = /etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem, ca = , trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true\r\n2018-04-17 23:50:11.431015 I | embed: listening for peers on https://0.0.0.0:2380\r\n2018-04-17 23:50:11.431144 I | embed: listening for client requests on 0.0.0.0:2379\r\n2018-04-17 23:50:11.434908 I | etcdserver: name = etcd-master\r\n2018-04-17 23:50:11.435002 I | etcdserver: data dir = /var/lib/rancher/etcd\r\n2018-04-17 23:50:11.435065 I | etcdserver: member dir = /var/lib/rancher/etcd/member\r\n2018-04-17 23:50:11.435156 I | etcdserver: heartbeat = 100ms\r\n2018-04-17 23:50:11.435230 I | etcdserver: election = 1000ms\r\n2018-04-17 23:50:11.435286 I | etcdserver: snapshot count = 100000\r\n2018-04-17 23:50:11.435371 I | etcdserver: advertise client URLs = https://127.0.0.1:2379,https://127.0.0.1:4001\r\n2018-04-17 23:50:11.435448 I | etcdserver: initial advertise peer URLs = https://127.0.0.1:2380\r\n2018-04-17 23:50:11.435512 I | etcdserver: initial cluster = etcd-master=https://127.0.0.1:2380\r\n2018-04-17 23:50:11.441342 I | etcdserver: starting member e92d66acd89ecf29 in cluster 7581d6eb2d25405b\r\n2018-04-17 23:50:11.441476 I | raft: e92d66acd89ecf29 became follower at term 0\r\n2018-04-17 23:50:11.441555 I | raft: newRaft e92d66acd89ecf29 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]\r\n2018-04-17 23:50:11.441640 I | raft: e92d66acd89ecf29 became follower at term 1\r\n2018/04/17 23:50:11 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r\n2018-04-17 23:50:11.453204 W | auth: simple token is not cryptographically signed\r\n2018-04-17 23:50:11.457386 I | etcdserver: starting server... [version: 3.2.13, cluster version: to_be_decided]\r\n2018-04-17 23:50:11.458169 I | embed: ClientTLS: cert = /etc/kubernetes/ssl/kube-etcd-127-0-0-1.pem, key = /etc/kubernetes/ssl/kube-etcd-127-0-0-1-key.pem, ca = , trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true\r\n2018-04-17 23:50:11.460462 I | etcdserver/membership: added member e92d66acd89ecf29 [https://127.0.0.1:2380] to cluster 7581d6eb2d25405b\r\n[restful] 2018/04/17 23:50:11 log.go:33: [restful/swagger] listing is available at https://10.43.0.1:6443/swaggerapi\r\n[restful] 2018/04/17 23:50:11 log.go:33: [restful/swagger] https://10.43.0.1:6443/swaggerui/ is mapped to folder /swagger-ui/\r\n2018-04-17 23:50:12.242100 I | raft: e92d66acd89ecf29 is starting a new election at term 1\r\n2018-04-17 23:50:12.242293 I | raft: e92d66acd89ecf29 became candidate at term 2\r\n2018-04-17 23:50:12.242368 I | raft: e92d66acd89ecf29 received MsgVoteResp from e92d66acd89ecf29 at term 2\r\n2018-04-17 23:50:12.242470 I | raft: e92d66acd89ecf29 became leader at term 2\r\n2018-04-17 23:50:12.242539 I | raft: raft.node: e92d66acd89ecf29 elected leader e92d66acd89ecf29 at term 2\r\n2018-04-17 23:50:12.243622 I | etcdserver: published {Name:etcd-master ClientURLs:[https://127.0.0.1:2379 https://127.0.0.1:4001]} to cluster 7581d6eb2d25405b\r\n2018-04-17 23:50:12.243761 I | etcdserver: setting up the initial cluster version to 3.2\r\n2018-04-17 23:50:12.244570 I | embed: ready to serve client requests\r\n2018-04-17 23:50:12.244847 I | embed: serving client requests on [::]:2379\r\n2018-04-17 23:50:12.246568 N | etcdserver/membership: set the initial cluster version to 3.2\r\n2018-04-17 23:50:12.246684 I | etcdserver/api: enabled capabilities for version 3.2\r\n[restful] 2018/04/17 23:50:12 log.go:33: [restful/swagger] listing is available at https://10.43.0.1:6443/swaggerapi\r\n[restful] 2018/04/17 23:50:12 log.go:33: [restful/swagger] https://10.43.0.1:6443/swaggerui/ is mapped to folder /swagger-ui/\r\n2018/04/17 23:50:13 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r\n2018/04/17 23:50:13 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r\n2018/04/17 23:50:13 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r\n2018/04/17 23:50:15 [INFO] Running kube-scheduler --v=2 --address=0.0.0.0 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true -v=1 --logtostderr=false --alsologtostderr=false\r\n2018/04/17 23:50:15 [INFO] Creating CRD authconfigs.management.cattle.io\r\n2018/04/17 23:50:15 [INFO] Creating CRD apps.project.cattle.io\r\nE0417 23:50:15.482177       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/server.go:103: Failed to list *v1.Pod: pods is forbidden: User \"system:kube-scheduler\" cannot list pods at the cluster scope\r\nE0417 23:50:15.490326       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: nodes is forbidden: User \"system:kube-scheduler\" cannot list nodes at the cluster scope\r\n2018/04/17 23:50:15 [INFO] Creating CRD clusteralerts.management.cattle.io\r\nE0417 23:50:15.492196       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User \"system:kube-scheduler\" cannot list persistentvolumes at the cluster scope\r\nE0417 23:50:15.492497       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User \"system:kube-scheduler\" cannot list replicasets.extensions at the cluster scope\r\nE0417 23:50:15.492679       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User \"system:kube-scheduler\" cannot list statefulsets.apps at the cluster scope\r\n2018/04/17 23:50:15 [INFO] Creating CRD namespacecomposeconfigs.project.cattle.io\r\nE0417 23:50:15.493260       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Service: services is forbidden: User \"system:kube-scheduler\" cannot list services at the cluster scope\r\nE0417 23:50:15.493887       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User \"system:kube-scheduler\" cannot list replicationcontrollers at the cluster scope\r\nE0417 23:50:15.493981       1 reflector.go:205] github.com/rancher/rancher/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User \"system:kube-scheduler\" cannot list persistentvolumeclaims at the cluster scope\r\n2018/04/17 23:50:15 [INFO] Running kube-controller-manager --configure-cloud-routes=false --enable-hostpath-provisioner=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --address=0.0.0.0 --allow-untagged-cloud=true --node-monitor-grace-period=40s --allocate-node-cidrs=true --cluster-cidr=10.42.0.0/16 --v=2 --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --leader-elect=true --pod-eviction-timeout=5m0s --service-cluster-ip-range=10.43.0.0/16 --service-account-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --use-service-account-credentials=true -v=1 --logtostderr=false --alsologtostderr=false\r\nFlag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release.  A cluster-id will be required on cloud instances\r\nE0417 23:50:15.495369       1 controllermanager.go:117] unable to register configz: register config \"componentconfig\" twice\r\n2018/04/17 23:50:15 [INFO] Creating CRD projectalerts.management.cattle.io\r\n2018/04/17 23:50:15 [INFO] Creating CRD notifiers.management.cattle.io\r\nE0417 23:50:15.513494       1 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: endpoints \"kube-controller-manager\" is forbidden: User \"system:kube-controller-manager\" cannot get endpoints in the namespace \"kube-system\"\r\n2018/04/17 23:50:15 [INFO] Waiting for CRD namespacecomposeconfigs.project.cattle.io to become available\r\n2018/04/17 23:50:15 [INFO] Creating CRD catalogs.management.cattle.io\r\n2018/04/17 23:50:15 [INFO] Creating CRD clusterevents.management.cattle.io\r\n2018/04/17 23:50:15 [INFO] Creating CRD clusterloggings.management.cattle.io\r\n2018/04/17 23:50:15 [INFO] Creating CRD clusterregistrationtokens.management.cattle.io\r\n2018/04/17 23:50:15 [INFO] Creating CRD clusterroletemplatebindings.management.cattle.io\r\n2018/04/17 23:50:16 [INFO] Creating CRD clusters.management.cattle.io\r\n2018/04/17 23:50:16 [INFO] Done waiting for CRD namespacecomposeconfigs.project.cattle.io to become available\r\n2018/04/17 23:50:16 [INFO] Creating CRD clustercomposeconfigs.management.cattle.io\r\n2018/04/17 23:50:16 [INFO] Creating CRD globalcomposeconfigs.management.cattle.io\r\n2018/04/17 23:50:16 [INFO] Creating CRD dynamicschemas.management.cattle.io\r\n2018/04/17 23:50:17 [INFO] Creating CRD globalrolebindings.management.cattle.io\r\n2018/04/17 23:50:17 [INFO] Creating CRD globalroles.management.cattle.io\r\n2018/04/17 23:50:17 [INFO] Creating CRD groupmembers.management.cattle.io\r\n2018/04/17 23:50:17 [INFO] Creating CRD groups.management.cattle.io\r\n2018/04/17 23:50:17 [INFO] Creating CRD listenconfigs.management.cattle.io\r\n2018/04/17 23:50:18 [INFO] Creating CRD nodes.management.cattle.io\r\n2018/04/17 23:50:18 [INFO] Creating CRD nodepools.management.cattle.io\r\n2018/04/17 23:50:18 [INFO] Creating CRD nodedrivers.management.cattle.io\r\n2018/04/17 23:50:18 [INFO] Creating CRD nodetemplates.management.cattle.io\r\n2018/04/17 23:50:18 [INFO] Creating CRD podsecuritypolicytemplates.management.cattle.io\r\n2018/04/17 23:50:19 [INFO] Creating CRD podsecuritypolicytemplateprojectbindings.management.cattle.io\r\n2018/04/17 23:50:19 [INFO] Creating CRD preferences.management.cattle.io\r\n2018/04/17 23:50:19 [INFO] Creating CRD projectloggings.management.cattle.io\r\n2018/04/17 23:50:19 [INFO] Creating CRD projectnetworkpolicies.management.cattle.io\r\n2018/04/17 23:50:19 [INFO] Creating CRD projectroletemplatebindings.management.cattle.io\r\n2018/04/17 23:50:20 [INFO] Creating CRD projects.management.cattle.io\r\n2018/04/17 23:50:20 [INFO] Creating CRD roletemplates.management.cattle.io\r\n2018/04/17 23:50:20 [INFO] Creating CRD settings.management.cattle.io\r\n2018/04/17 23:50:20 [INFO] Creating CRD templates.management.cattle.io\r\n2018/04/17 23:50:20 [INFO] Creating CRD templateversions.management.cattle.io\r\nE0417 23:50:20.950487       1 garbagecollector.go:113] failed to sync all monitors: [couldn't look up resource {\"management.cattle.io\" \"v3\" \"roletemplates\"}: no matches for {management.cattle.io v3 roletemplates}, couldn't look up resource {\"management.cattle.io\" \"v3\" \"settings\"}: no matches for {management.cattle.io v3 settings}, couldn't look up resource {\"management.cattle.io\" \"v3\" \"templates\"}: no matches for {management.cattle.io v3 templates}, couldn't look up resource {\"management.cattle.io\" \"v3\" \"projectroletemplatebindings\"}: no matches for {management.cattle.io v3 projectroletemplatebindings}, couldn't look up resource {\"management.cattle.io\" \"v3\" \"projects\"}: no matches for {management.cattle.io v3 projects}]\r\n2018/04/17 23:50:21 [INFO] Creating CRD clusterpipelines.management.cattle.io\r\n2018/04/17 23:50:21 [INFO] Creating CRD pipelines.management.cattle.io\r\n2018/04/17 23:50:21 [INFO] Creating CRD pipelineexecutions.management.cattle.io\r\n2018/04/17 23:50:21 [INFO] Creating CRD pipelineexecutionlogs.management.cattle.io\r\n2018/04/17 23:50:21 [INFO] Creating CRD sourcecodecredentials.management.cattle.io\r\n2018/04/17 23:50:22 [INFO] Creating CRD sourcecoderepositories.management.cattle.io\r\nE0417 23:50:22.147876       1 certificates.go:48] Failed to start certificate controller: error reading CA cert file \"/etc/kubernetes/ca/ca.pem\": open /etc/kubernetes/ca/ca.pem: no such file or directory\r\n2018/04/17 23:50:22 [INFO] Creating CRD tokens.management.cattle.io\r\n2018/04/17 23:50:22 [INFO] Creating CRD users.management.cattle.io\r\n2018/04/17 23:50:22 [INFO] Starting API controllers\r\n2018/04/17 23:50:22 [INFO] Syncing SecretController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ListenConfigController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing RoleController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ProjectController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ClusterRegistrationTokenController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ClusterController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing DynamicSchemaController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing NodeController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing NodeController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing UserController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ClusterRoleTemplateBindingController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ProjectRoleTemplateBindingController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ClusterRoleBindingController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing RoleBindingController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ClusterRoleController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing ClusterPipelineController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing SourceCodeCredentialController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing SourceCodeRepositoryController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing PipelineController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing GroupMemberController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing NodeDriverController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing TokenController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing GroupController Controller\r\n2018/04/17 23:50:22 [INFO] Syncing SettingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing SecretController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ListenConfigController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing RoleController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing SettingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing DynamicSchemaController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing RoleBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRegistrationTokenController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NodeController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing UserController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NodeController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleTemplateBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectRoleTemplateBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing SourceCodeCredentialController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterPipelineController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing PipelineController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing SourceCodeRepositoryController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing GroupMemberController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing TokenController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NodeDriverController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing GroupController Controller Done\r\n2018/04/17 23:50:23 [INFO] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"kube-system\", Name:\"cattle-controllers\", UID:\"18228296-429a-11e8-bc4b-0242ac110002\", APIVersion:\"v1\", ResourceVersion:\"275\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' 2cc2a9fc81d2 became leader\r\n2018/04/17 23:50:23 [INFO] Starting catalog controller\r\n2018/04/17 23:50:23 [INFO] Starting management controllers\r\n2018/04/17 23:50:23 [INFO] Syncing SecretController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing PodSecurityPolicyTemplateProjectBindingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing TokenController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing UserController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing CatalogController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterEventController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing GlobalRoleBindingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing NodeController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing PodSecurityPolicyTemplateController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing GlobalComposeConfigController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing DynamicSchemaController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing NodeDriverController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing NodePoolController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing RoleController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing NamespaceController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing RoleTemplateController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing GlobalRoleController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectRoleTemplateBindingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleTemplateBindingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectAlertController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleBindingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing RoleBindingController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleController Controller\r\n2018/04/17 23:50:23 [INFO] Syncing SecretController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing TokenController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing UserController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NodeController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NodeDriverController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing RoleController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NamespaceController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectRoleTemplateBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleTemplateBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing RoleBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing DynamicSchemaController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing PodSecurityPolicyTemplateProjectBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterEventController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing CatalogController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing GlobalRoleBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing NodePoolController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing PodSecurityPolicyTemplateController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing RoleTemplateController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing GlobalRoleController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ProjectAlertController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing GlobalComposeConfigController Controller Done\r\n2018/04/17 23:50:23 [INFO] Syncing ClusterRoleBindingController Controller Done\r\n2018/04/17 23:50:23 [INFO] Reconciling GlobalRoles\r\n2018/04/17 23:50:23 [INFO] Creating podsecuritypolicytemplates-manage\r\n2018/04/17 23:50:23 [INFO] Listening on :443\r\n2018/04/17 23:50:23 [INFO] Listening on :80\r\n2018/04/17 23:50:23 [INFO] Creating user\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-podsecuritypolicytemplates-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating nodedrivers-manage\r\n2018/04/17 23:50:23 [INFO] Creating users-manage\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-user for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-nodedrivers-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating authn-manage\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-users-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating roles-manage\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-authn-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating settings-manage\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-roles-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating admin\r\n2018/04/17 23:50:23 [INFO] Creating user-base\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-settings-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating clusters-create\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-admin for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-user-base for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating catalogs-manage\r\n2018/04/17 23:50:23 [INFO] Creating catalogs-use\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-clusters-create for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-catalogs-manage for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Reconciling RoleTemplates\r\n2018/04/17 23:50:23 [INFO] Creating cluster-admin\r\n2018/04/17 23:50:23 [INFO] Creating new ClusterRole cattle-globalrole-catalogs-use for corresponding GlobalRole\r\n2018/04/17 23:50:23 [INFO] Creating admin\r\n2018/04/17 23:50:24 [INFO] Creating edit\r\n2018/04/17 23:50:24 [INFO] Creating cluster-member\r\n2018/04/17 23:50:24 [INFO] Creating create-ns\r\n2018/04/17 23:50:24 [INFO] Creating secrets-manage\r\n2018/04/17 23:50:24 [INFO] Creating workloads-view\r\n2018/04/17 23:50:24 [INFO] Creating serviceaccounts-manage\r\n2018/04/17 23:50:24 [INFO] Creating view\r\n2018/04/17 23:50:24 [INFO] Creating projects-view\r\n2018/04/17 23:50:24 [INFO] Creating clusterroletemplatebindings-view\r\n2018/04/17 23:50:24 [INFO] Creating workloads-manage\r\n2018/04/17 23:50:24 [INFO] Creating secrets-view\r\n2018/04/17 23:50:24 [INFO] Creating projectroletemplatebindings-view\r\n2018/04/17 23:50:24 [INFO] Creating nodes-manage\r\n2018/04/17 23:50:24 [INFO] Creating storage-manage\r\n2018/04/17 23:50:24 [INFO] Creating read-only\r\n2018/04/17 23:50:24 [INFO] Creating configmaps-view\r\n2018/04/17 23:50:24 [INFO] Creating persistentvolumeclaims-view\r\n2018/04/17 23:50:24 [INFO] Creating serviceaccounts-view\r\n2018/04/17 23:50:24 [INFO] Creating persistentvolumeclaims-manage\r\n2018/04/17 23:50:24 [INFO] Creating projectroletemplatebindings-manage\r\n2018/04/17 23:50:24 [INFO] Creating projects-create\r\n2018/04/17 23:50:24 [INFO] Creating nodes-view\r\n2018/04/17 23:50:24 [INFO] Creating clusterroletemplatebindings-manage\r\n2018/04/17 23:50:24 [INFO] Creating project-owner\r\n2018/04/17 23:50:24 [INFO] Creating ingress-manage\r\n2018/04/17 23:50:24 [INFO] Creating services-manage\r\n2018/04/17 23:50:24 [INFO] Creating cluster-owner\r\n2018/04/17 23:50:24 [INFO] Creating configmaps-manage\r\n2018/04/17 23:50:24 [INFO] Creating project-member\r\n2018/04/17 23:50:24 [INFO] Creating ingress-view\r\n2018/04/17 23:50:24 [INFO] Creating services-view\r\n2018/04/17 23:50:24 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding globalrolebinding-2kp42\r\n2018/04/17 23:50:24 [INFO] Creating node driver amazonec2\r\n2018/04/17 23:50:24 [INFO] Creating node driver azure\r\n2018/04/17 23:50:24 [INFO] Creating node driver digitalocean\r\n2018/04/17 23:50:24 [INFO] Creating node driver exoscale\r\n2018/04/17 23:50:24 [INFO] Creating node driver openstack\r\n2018/04/17 23:50:24 [INFO] Creating node driver otc\r\n2018/04/17 23:50:24 [INFO] Creating node driver packet\r\n2018/04/17 23:50:24 [INFO] Creating node driver rackspace\r\n2018/04/17 23:50:24 [INFO] Creating node driver softlayer\r\n2018/04/17 23:50:24 [INFO] Creating node driver vmwarevsphere\r\nE0417 23:50:24.375591       1 generic_controller.go:204] NodeDriverController openstack [node-driver-controller] failed with : dynamicschemas.management.cattle.io \"nodetemplateconfig\" already exists\r\n2018/04/17 23:50:24 [INFO] uploading azureConfig to node schema\r\n2018/04/17 23:50:24 [INFO] uploading azureConfig to node schema\r\n2018/04/17 23:50:24 [INFO] uploading amazonec2Config to node schema\r\n2018/04/17 23:50:24 [INFO] uploading amazonec2Config to node schema\r\n2018/04/17 23:50:24 [INFO] uploading digitaloceanConfig to node schema\r\n2018/04/17 23:50:24 [INFO] uploading digitaloceanConfig to node schema\r\n2018/04/17 23:50:24 [INFO] uploading vmwarevsphereConfig to node schema\r\n2018/04/17 23:50:24 [INFO] uploading vmwarevsphereConfig to node schema\r\nE0417 23:50:32.752523       1 core.go:70] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail.\r\n2018/04/17 23:55:37 [INFO] stdout: Creating CA: management-state/node/nodes/rancher-node1/certs/ca.pem\r\n2018/04/17 23:55:38 [INFO] stdout: Creating client certificate: management-state/node/nodes/rancher-node1/certs/cert.pem\r\n2018/04/17 23:55:38 [INFO] stdout: Running pre-create checks...\r\n2018/04/17 23:55:38 [INFO] stdout: Creating machine...\r\n2018/04/17 23:55:38 [INFO] stdout: (rancher-node1) Launching instance...\r\n2018/04/17 23:55:56 [INFO] stdout: Waiting for machine to be running, this may take a few minutes...\r\n2018/04/17 23:55:56 [INFO] stdout: Detecting operating system of created instance...\r\n2018/04/17 23:55:56 [INFO] stdout: Waiting for SSH to be available...\r\n2018/04/17 23:56:05 [INFO] stdout: Detecting the provisioner...\r\n2018/04/17 23:56:05 [INFO] stdout: Provisioning with ubuntu(systemd)...\r\n2018/04/17 23:56:16 [INFO] stdout: Installing Docker...\r\n2018/04/17 23:56:31 [INFO] stdout: Copying certs to the local machine directory...\r\n2018/04/17 23:56:33 [INFO] stdout: Copying certs to the remote machine...\r\n2018/04/17 23:56:34 [INFO] stdout: Setting Docker configuration on the remote daemon...\r\n2018/04/17 23:56:37 [INFO] stdout: Checking connection to Docker...\r\n2018/04/17 23:56:37 [INFO] stdout: Docker is up and running!\r\n2018/04/17 23:56:43 [INFO] Generating and uploading node config rancher-node1\r\n2018/04/17 23:56:43 [INFO] Provisioning cluster [cluster-w4xcv]\r\n2018/04/17 23:56:43 [INFO] Creating cluster [cluster-w4xcv]\r\n2018/04/17 23:56:43 [INFO] cluster [cluster-w4xcv] provisioning: Building Kubernetes cluster\r\n2018/04/17 23:56:43 [INFO] cluster [cluster-w4xcv] provisioning: [dialer] Setup tunnel for host [172.22.1.213]\r\n2018/04/17 23:56:43 [INFO] cluster [cluster-w4xcv] provisioning: [network] Deploying port listener containers\r\n2018/04/17 23:56:43 [INFO] cluster [cluster-w4xcv] provisioning: [network] Pulling image [alpine:latest] on host [172.22.1.213]\r\n2018/04/17 23:56:45 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully pulled image [alpine:latest] on host [172.22.1.213]\r\n2018/04/17 23:56:45 [INFO] Handling backend connection request.\r\n2018/04/17 23:56:45 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully started [rke-etcd-port-listener] container on host [172.22.1.213]\r\n2018/04/17 23:56:45 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully started [rke-cp-port-listener] container on host [172.22.1.213]\r\n2018/04/17 23:56:45 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully started [rke-worker-port-listener] container on host [172.22.1.213]\r\n2018/04/17 23:56:45 [INFO] cluster [cluster-w4xcv] provisioning: [network] Port listener containers deployed successfully\r\n2018/04/17 23:56:45 [INFO] cluster [cluster-w4xcv] provisioning: [network] Running control plane -> etcd port checks\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully started [rke-port-checker] container on host [172.22.1.213]\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [network] Running control plane -> worker port checks\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully started [rke-port-checker] container on host [172.22.1.213]\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [network] Running workers -> control plane port checks\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [network] Successfully started [rke-port-checker] container on host [172.22.1.213]\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [network] Removing port listener containers\r\n2018/04/17 23:56:46 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-etcd-port-listener] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-cp-port-listener] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-worker-port-listener] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [network] Port listener containers removed successfully\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Attempting to recover certificates from backup on host [172.22.1.213]\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Successfully started [cert-fetcher] container on host [172.22.1.213]\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] No Certificate backup found on host [172.22.1.213]\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating CA kubernetes certificates\r\n2018/04/17 23:56:47 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating Kubernetes API server certificates\r\n2018/04/17 23:56:48 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating Kube Controller certificates\r\n2018/04/17 23:56:48 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating Kube Scheduler certificates\r\n2018/04/17 23:56:49 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating Kube Proxy certificates\r\n2018/04/17 23:56:49 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating Node certificate\r\n2018/04/17 23:56:49 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating admin certificates and kubeconfig\r\n2018/04/17 23:56:49 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Generating etcd-172.22.1.213 certificate and key\r\n2018/04/17 23:56:50 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Temporarily saving certs to control host [172.22.1.213]\r\n2018/04/17 23:56:50 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Pulling image [rancher/rke-cert-deployer:v0.1.1] on host [172.22.1.213]\r\n2018/04/17 23:56:52 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Successfully pulled image [rancher/rke-cert-deployer:v0.1.1] on host [172.22.1.213]\r\n2018/04/17 23:56:58 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Saved certs to control host [172.22.1.213]\r\n2018/04/17 23:56:58 [INFO] cluster [cluster-w4xcv] provisioning: [reconcile] Reconciling cluster state\r\n2018/04/17 23:56:58 [INFO] cluster [cluster-w4xcv] provisioning: [reconcile] This is newly generated cluster\r\n2018/04/17 23:56:58 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Deploying kubernetes certificates to Cluster nodes\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: Successfully Deployed local admin kubeconfig at [management-state/rke/rke-249470891/kube_config_cluster.yml]\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Successfully deployed kubernetes certificates to Cluster nodes\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [cloud] Deploying cloud config file to node [172.22.1.213]\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [cloud] Successfully started [cloud-config-deployer] container on host [172.22.1.213]\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [remove/cloud-config-deployer] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [cloud] Successfully deployed kubernetes cloud config to Cluster nodes\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: Pre-pulling kubernetes images\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: Kubernetes images pulled successfully\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [etcd] Building up etcd plane..\r\n2018/04/17 23:57:03 [INFO] cluster [cluster-w4xcv] provisioning: [etcd] Pulling image [rancher/coreos-etcd:v3.0.17] on host [172.22.1.213]\r\n2018/04/17 23:57:04 [INFO] cluster [cluster-w4xcv] provisioning: [etcd] Successfully pulled image [rancher/coreos-etcd:v3.0.17] on host [172.22.1.213]\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [etcd] Successfully started [etcd] container on host [172.22.1.213]\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [etcd] Successfully started [rke-log-linker] container on host [172.22.1.213]\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-log-linker] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [etcd] Successfully started etcd plane..\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Building up Controller Plane..\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [sidekick] Pulling image [rancher/rke-service-sidekick:v0.1.2] on host [172.22.1.213]\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [sidekick] Successfully pulled image [rancher/rke-service-sidekick:v0.1.2] on host [172.22.1.213]\r\n2018/04/17 23:57:05 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Pulling image [docker.io/rancher/k8s:v1.8.10-rancher1-1] on host [172.22.1.213]\r\nE0417 23:57:17.127739       1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus\r\n2018/04/17 23:57:46 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully pulled image [docker.io/rancher/k8s:v1.8.10-rancher1-1] on host [172.22.1.213]\r\n2018/04/17 23:57:47 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started [kube-apiserver] container on host [172.22.1.213]\r\n2018/04/17 23:57:47 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] Start Healthcheck on service [kube-apiserver] on host [172.22.1.213]\r\n2018/04/17 23:57:57 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] service [kube-apiserver] on host [172.22.1.213] is healthy\r\n2018/04/17 23:57:57 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started [rke-log-linker] container on host [172.22.1.213]\r\n2018/04/17 23:57:57 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-log-linker] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:57:57 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started [kube-controller-manager] container on host [172.22.1.213]\r\n2018/04/17 23:57:57 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [172.22.1.213]\r\n2018/04/17 23:58:02 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] service [kube-controller-manager] on host [172.22.1.213] is healthy\r\n2018/04/17 23:58:03 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started [rke-log-linker] container on host [172.22.1.213]\r\n2018/04/17 23:58:03 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-log-linker] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:58:03 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started [kube-scheduler] container on host [172.22.1.213]\r\n2018/04/17 23:58:03 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] Start Healthcheck on service [kube-scheduler] on host [172.22.1.213]\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] service [kube-scheduler] on host [172.22.1.213] is healthy\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started [rke-log-linker] container on host [172.22.1.213]\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-log-linker] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [controlplane] Successfully started Controller Plane..\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [authz] Creating rke-job-deployer ServiceAccount\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [authz] rke-job-deployer ServiceAccount created successfully\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [authz] Creating system:node ClusterRoleBinding\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [authz] system:node ClusterRoleBinding created successfully\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Save kubernetes certificates as secrets\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [certificates] Successfully saved certificates as kubernetes secret [k8s-certs]\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [state] Saving cluster state to Kubernetes\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [worker] Building up Worker Plane..\r\n2018/04/17 23:58:08 [INFO] cluster [cluster-w4xcv] provisioning: [sidekick] Sidekick container already created on host [172.22.1.213]\r\n2018/04/17 23:58:09 [INFO] cluster [cluster-w4xcv] provisioning: [worker] Successfully started [kubelet] container on host [172.22.1.213]\r\n2018/04/17 23:58:09 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] Start Healthcheck on service [kubelet] on host [172.22.1.213]\r\n2018/04/17 23:58:14 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] service [kubelet] on host [172.22.1.213] is healthy\r\n2018/04/17 23:58:14 [INFO] cluster [cluster-w4xcv] provisioning: [worker] Successfully started [rke-log-linker] container on host [172.22.1.213]\r\n2018/04/17 23:58:14 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-log-linker] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:58:14 [INFO] cluster [cluster-w4xcv] provisioning: [worker] Successfully started [kube-proxy] container on host [172.22.1.213]\r\n2018/04/17 23:58:14 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.22.1.213]\r\n2018/04/17 23:58:19 [INFO] cluster [cluster-w4xcv] provisioning: [healthcheck] service [kube-proxy] on host [172.22.1.213] is healthy\r\n2018/04/17 23:58:19 [INFO] cluster [cluster-w4xcv] provisioning: [worker] Successfully started [rke-log-linker] container on host [172.22.1.213]\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [remove/rke-log-linker] Successfully removed container on host [172.22.1.213]\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [worker] Successfully started Worker Plane..\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [sync] Syncing nodes Labels and Taints\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [sync] Successfully synced nodes Labels and Taints\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [network] Setting up network plugin: canal\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Saving addon ConfigMap to Kubernetes\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin\r\n2018/04/17 23:58:20 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Executing deploy job..\r\n2018/04/17 23:58:25 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Setting up KubeDNS\r\n2018/04/17 23:58:25 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Saving addon ConfigMap to Kubernetes\r\n2018/04/17 23:58:25 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon\r\n2018/04/17 23:58:25 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Executing deploy job..\r\n2018/04/17 23:58:30 [INFO] cluster [cluster-w4xcv] provisioning: [addons] KubeDNS deployed successfully..\r\n2018/04/17 23:58:30 [INFO] cluster [cluster-w4xcv] provisioning: [ingress] Setting up nginx ingress controller\r\n2018/04/17 23:58:30 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Saving addon ConfigMap to Kubernetes\r\n2018/04/17 23:58:30 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller\r\n2018/04/17 23:58:30 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Executing deploy job..\r\n2018/04/17 23:58:45 [INFO] cluster [cluster-w4xcv] provisioning: [ingress] ingress controller nginx is successfully deployed\r\n2018/04/17 23:58:45 [INFO] cluster [cluster-w4xcv] provisioning: [addons] Setting up user addons\r\n2018/04/17 23:58:45 [INFO] cluster [cluster-w4xcv] provisioning: [addons] no user addons defined\r\n2018/04/17 23:58:45 [INFO] cluster [cluster-w4xcv] provisioning: Finished building Kubernetes cluster successfully\r\n2018/04/17 23:58:45 [INFO] Provisioned cluster [cluster-w4xcv]\r\n2018/04/17 23:58:45 [INFO] Starting cluster agent forcluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding globalrolebinding-h8qjk\r\n2018/04/17 23:58:45 [INFO] Registering project network policy\r\n2018/04/17 23:58:45 [INFO] Registering namespaceHandler for adding labels \r\n2018/04/17 23:58:45 [INFO] registering podsecuritypolicy cluster handler for cluster cluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] registering podsecuritypolicy project handler for cluster cluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] registering podsecuritypolicy namespace handler for cluster cluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] registering podsecuritypolicy serviceaccount handler for cluster cluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] registering podsecuritypolicy template handler for cluster cluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] Starting cluster controllers for cluster-w4xcv\r\n2018/04/17 23:58:45 [INFO] Syncing NetworkPolicyController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterComposeConfigController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NamespaceComposeConfigController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NamespaceController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing StatefulSetController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing EndpointsController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing RoleController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing IngressController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing JobController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing CronJobController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterAlertController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectAlertController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NotifierController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NotifierController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NodeController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectRoleTemplateBindingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterRoleTemplateBindingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing RoleTemplateController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing GlobalRoleBindingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterLoggingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectLoggingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterLoggingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectNetworkPolicyController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PipelineController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PipelineExecutionController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PipelineExecutionLogController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing SourceCodeCredentialController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing SourceCodeRepositoryController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterPipelineController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PodSecurityPolicyTemplateController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PodSecurityPolicyTemplateProjectBindingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing AppController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing SecretController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing DeploymentController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ReplicaSetController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing DaemonSetController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ServiceController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PodController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NodeController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ReplicationControllerController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing PodSecurityPolicyController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing EventController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NamespaceController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing SecretController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ServiceAccountController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterRoleBindingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing RoleBindingController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterRoleController Controller\r\n2018/04/17 23:58:45 [INFO] Syncing NetworkPolicyController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterComposeConfigController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NamespaceComposeConfigController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NamespaceController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing StatefulSetController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing EndpointsController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing RoleController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NotifierController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing IngressController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing JobController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing CronJobController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NodeController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PipelineController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PipelineExecutionController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ReplicaSetController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PodSecurityPolicyController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ServiceAccountController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterRoleBindingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing RoleBindingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterRoleController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterAlertController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectAlertController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectRoleTemplateBindingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterRoleTemplateBindingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing RoleTemplateController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterLoggingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectNetworkPolicyController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PipelineExecutionLogController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing SourceCodeCredentialController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterPipelineController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing DeploymentController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing DaemonSetController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ServiceController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PodController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NodeController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ReplicationControllerController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NamespaceController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing SecretController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing NotifierController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing GlobalRoleBindingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ProjectLoggingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing ClusterLoggingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing SourceCodeRepositoryController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PodSecurityPolicyTemplateProjectBindingController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing PodSecurityPolicyTemplateController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing AppController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing SecretController Controller Done\r\n2018/04/17 23:58:45 [INFO] Syncing EventController Controller Done\r\n2018/04/17 23:58:46 [ERROR] error updating ns kube-public status: Operation cannot be fulfilled on namespaces \"kube-public\": the object has been modified; please apply your changes to the latest version and try again\r\n2018/04/17 23:58:46 [ERROR] error updating ns ingress-nginx status: Operation cannot be fulfilled on namespaces \"ingress-nginx\": the object has been modified; please apply your changes to the latest version and try again\r\n2018/04/17 23:58:46 [ERROR] error updating ns kube-system status: Operation cannot be fulfilled on namespaces \"kube-system\": the object has been modified; please apply your changes to the latest version and try again\r\n2018/04/17 23:58:46 [INFO] Created machine for node [ip-172-22-1-213.ec2.internal]\r\nE0417 23:58:46.257157       1 generic_controller.go:204] ClusterRoleTemplateBindingController cluster-w4xcv/creator [cluster-crtb-sync] failed with : couldn't ensure roles: couldn't create role cluster-owner: clusterroles.rbac.authorization.k8s.io \"cluster-owner\" already exists\r\n2018/04/17 23:58:46 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=project-9tmr4 to namespace=default\r\n2018/04/17 23:58:46 [INFO] projectSyncer: createDefaultNetworkPolicy: successfully created default network policy for project: project-9tmr4\r\n2018/04/17 23:58:46 [ERROR] error updating ns cattle-system status: Operation cannot be fulfilled on namespaces \"cattle-system\": the object has been modified; please apply your changes to the latest version and try again\r\n2018/04/17 23:58:47 [INFO] Updating workload [ingress-nginx/nginx-ingress-controller] with public endpoints [[{\"nodeName\":\"cluster-w4xcv:machine-8n4mw\",\"addresses\":[\"172.22.1.213\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"ingress-nginx:nginx-ingress-controller-74cvf\",\"allNodes\":false},{\"nodeName\":\"cluster-w4xcv:machine-8n4mw\",\"addresses\":[\"172.22.1.213\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"ingress-nginx:nginx-ingress-controller-74cvf\",\"allNodes\":false}]]\r\n2018/04/17 23:58:47 [INFO] Updating pod [ingress-nginx/nginx-ingress-controller-74cvf] with public endpoints [[{\"nodeName\":\"cluster-w4xcv:machine-8n4mw\",\"addresses\":[\"172.22.1.213\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"ingress-nginx:nginx-ingress-controller-74cvf\",\"allNodes\":false},{\"nodeName\":\"cluster-w4xcv:machine-8n4mw\",\"addresses\":[\"172.22.1.213\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"ingress-nginx:nginx-ingress-controller-74cvf\",\"allNodes\":false}]]\r\n2018/04/17 23:58:47 [INFO] Updating node [ip-172-22-1-213.ec2.internal] with public endpoints [[{\"nodeName\":\"cluster-w4xcv:machine-8n4mw\",\"addresses\":[\"172.22.1.213\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"ingress-nginx:nginx-ingress-controller-74cvf\",\"allNodes\":false},{\"nodeName\":\"cluster-w4xcv:machine-8n4mw\",\"addresses\":[\"172.22.1.213\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"ingress-nginx:nginx-ingress-controller-74cvf\",\"allNodes\":false}]]\r\n2018/04/17 23:58:48 [INFO] Handling backend connection request.\r\n2018/04/17 23:58:48 [INFO] Handling backend connection request.\r\n2018-04-18 00:00:11.552907 I | mvcc: store.index: compact 903\r\n2018-04-18 00:00:11.554332 I | mvcc: finished scheduled compaction at 903 (took 1.02128ms)\r\n2018-04-18 00:05:11.557985 I | mvcc: store.index: compact 1691\r\n2018-04-18 00:05:11.559851 I | mvcc: finished scheduled compaction at 1691 (took 1.298437ms)\r\n2018-04-18 00:10:11.563449 I | mvcc: store.index: compact 2143\r\n2018-04-18 00:10:11.564509 I | mvcc: finished scheduled compaction at 2143 (took 665.342µs)\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/382210481","html_url":"https://github.com/rancher/rancher/issues/12809#issuecomment-382210481","issue_url":"https://api.github.com/repos/rancher/rancher/issues/12809","id":382210481,"user":{"login":"mmorianos","id":4206393,"avatar_url":"https://avatars2.githubusercontent.com/u/4206393?v=4","gravatar_id":"","url":"https://api.github.com/users/mmorianos","html_url":"https://github.com/mmorianos","followers_url":"https://api.github.com/users/mmorianos/followers","following_url":"https://api.github.com/users/mmorianos/following{/other_user}","gists_url":"https://api.github.com/users/mmorianos/gists{/gist_id}","starred_url":"https://api.github.com/users/mmorianos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mmorianos/subscriptions","organizations_url":"https://api.github.com/users/mmorianos/orgs","repos_url":"https://api.github.com/users/mmorianos/repos","events_url":"https://api.github.com/users/mmorianos/events{/privacy}","received_events_url":"https://api.github.com/users/mmorianos/received_events","type":"User","site_admin":false},"created_at":"2018-04-18T01:10:19Z","updated_at":"2018-04-18T01:10:19Z","author_association":"NONE","body":"a bit more information.  the above deployment is as follows.  The rancher/server container is running on a Ubuntu 16.04.1 LTS instance, in a VPC that has public/private subnets.  The EC2 instance is on an public subnet and is reachable via it's public IP/hostname.  In the node template I put the rancher-nodes (either a single node with all roles, or multiple nodes each running either etcd, control plance, or worker roles) on private subnets, in the same VPC.  The Rancher Server URL is set to the internal IP address of the instance running rancher/server.  Security-group allows all IP's all Ports to talk to one another in the same security-group and all EC2 instances are members of this group.\r\n\r\nLet me know if I need to provide more details on the account setup.  I imagine it would be pretty common where a customer does not want to have any inter cluster communication occurring over public channels."}}
{"actor":{"display_login":"mmorianos"}}