{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/13382","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/13382/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/13382/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/13382/events","html_url":"https://github.com/rancher/rancher/issues/13382","id":321542570,"node_id":"MDU6SXNzdWUzMjE1NDI1NzA=","number":13382,"title":"Worker nodes randomly become inactive","user":{"login":"MariusDieckmann","id":13437264,"node_id":"MDQ6VXNlcjEzNDM3MjY0","avatar_url":"https://avatars0.githubusercontent.com/u/13437264?v=4","gravatar_id":"","url":"https://api.github.com/users/MariusDieckmann","html_url":"https://github.com/MariusDieckmann","followers_url":"https://api.github.com/users/MariusDieckmann/followers","following_url":"https://api.github.com/users/MariusDieckmann/following{/other_user}","gists_url":"https://api.github.com/users/MariusDieckmann/gists{/gist_id}","starred_url":"https://api.github.com/users/MariusDieckmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MariusDieckmann/subscriptions","organizations_url":"https://api.github.com/users/MariusDieckmann/orgs","repos_url":"https://api.github.com/users/MariusDieckmann/repos","events_url":"https://api.github.com/users/MariusDieckmann/events{/privacy}","received_events_url":"https://api.github.com/users/MariusDieckmann/received_events","type":"User","site_admin":false},"labels":[{"id":601597122,"node_id":"MDU6TGFiZWw2MDE1OTcxMjI=","url":"https://api.github.com/repos/rancher/rancher/labels/status/more-info","name":"status/more-info","color":"bfdadc","default":false},{"id":659821934,"node_id":"MDU6TGFiZWw2NTk4MjE5MzQ=","url":"https://api.github.com/repos/rancher/rancher/labels/version/2.0","name":"version/2.0","color":"e99695","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-09T12:02:22Z","updated_at":"2018-07-23T19:03:56Z","closed_at":null,"author_association":"NONE","body":"**Rancher versions:**\r\nrancher/server or rancher/rancher: 2.0.0\r\nrancher/agent or rancher/rancher-agent: 2.0.0\r\n\r\n**Infrastructure Stack versions:**\r\nNAME=\"RancherOS\"\r\nVERSION=v1.3.0\r\nID=rancheros\r\nID_LIKE=\r\nVERSION_ID=v1.3.0\r\nPRETTY_NAME=\"RancherOS v1.3.0\"\r\nHOME_URL=\"http://rancher.com/rancher-os/\"\r\nSUPPORT_URL=\"https://forums.rancher.com/c/rancher-os\"\r\nBUG_REPORT_URL=\"https://github.com/rancher/os/issues\"\r\nBUILD_ID=\r\n\r\n**Docker version: (`docker version`,`docker info` preferred)**\r\nContainers: 14\r\n Running: 11\r\n Paused: 0\r\n Stopped: 3\r\nImages: 8\r\nServer Version: 17.03.2-ce\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins: \r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 4ab9917febca54791c5f071a9d1f404867857fcc\r\nrunc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe\r\ninit version: 949e6fa\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.9.80-rancher\r\nOperating System: RancherOS v1.3.0\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 16\r\nTotal Memory: 31.42 GiB\r\nName: kubeworker-4.novalocal\r\nID: 3EF3:B22K:3QQ3:TWH4:J6J5:5E22:7ZI4:YUFY:LS36:KR7V:SOEG:SI5W\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\n**Operating system and kernel: (`cat /etc/os-release`, `uname -r` preferred)**\r\n4.9.80-rancher\r\n**Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)**\r\nOpenstack\r\n**Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)**\r\nsingle node rancher, installed via VM\r\n**Environment Template: (Cattle/Kubernetes/Swarm/Mesos)**\r\nKubernetes\r\n**Steps to Reproduce:**\r\nProvision VMs in Openstack with RancherOS 1.3.0, same error occured with Ubuntu 16.04\r\nInstall and add custom cluster using the docker command presented in the interface. Deploy cluster, wait.\r\n[dockerpslog.log](https://github.com/rancher/rancher/files/1987679/dockerpslog.log)\r\n\r\n**Node system information**\r\n\r\nArchitecture | amd64\r\n-- | --\r\nDocker Version | 17.3.2\r\nKernel Version | 4.9.80-rancher\r\nKubelet Version | v1.10.1\r\nKube Proxy Version | v1.10.1\r\nOperating System Image | Debian GNU/Linux 9 (stretch)\r\nOperating System | linux\r\n\r\n**Results:**\r\nIndividual worker nodes seem to stop working with error message: Kubelet stopped posting node status.\r\nThe information placed above is directly taken from one of these nodes. The error only occurs occasionally and with no apparent reason. They become active again after some time.\r\nThe nodes remain inactive in this state for a couple of minutes, maybe up to 10.\r\nFor further details like logs from single docker container on inactive nodes just ask.\r\n\r\nThank in advance\r\nMarius"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/407166607","html_url":"https://github.com/rancher/rancher/issues/13382#issuecomment-407166607","issue_url":"https://api.github.com/repos/rancher/rancher/issues/13382","id":407166607,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzE2NjYwNw==","user":{"login":"MariusDieckmann","id":13437264,"node_id":"MDQ6VXNlcjEzNDM3MjY0","avatar_url":"https://avatars0.githubusercontent.com/u/13437264?v=4","gravatar_id":"","url":"https://api.github.com/users/MariusDieckmann","html_url":"https://github.com/MariusDieckmann","followers_url":"https://api.github.com/users/MariusDieckmann/followers","following_url":"https://api.github.com/users/MariusDieckmann/following{/other_user}","gists_url":"https://api.github.com/users/MariusDieckmann/gists{/gist_id}","starred_url":"https://api.github.com/users/MariusDieckmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MariusDieckmann/subscriptions","organizations_url":"https://api.github.com/users/MariusDieckmann/orgs","repos_url":"https://api.github.com/users/MariusDieckmann/repos","events_url":"https://api.github.com/users/MariusDieckmann/events{/privacy}","received_events_url":"https://api.github.com/users/MariusDieckmann/received_events","type":"User","site_admin":false},"created_at":"2018-07-23T19:03:56Z","updated_at":"2018-07-23T19:03:56Z","author_association":"NONE","body":"In our case this issue was based on issues with the underlaying network infrastructure, we fixed it by fixing our routers. Here is what i think made this problem occur. The TCP heartbeat connection from the kubelet became unroutable and was stuck. The kubelet however has a long timeout on this connection and will only try to reconnect after 15 minutes, in the meanwhile the kubelet will be reported unavailable. There is a fix available in 1.11 which will be backported to older versions according to this: https://github.com/kubernetes/kubernetes/pull/63492\r\n\r\nYou can verfify this by checking for how long the kubelet becomes inactive. If its 15 minutes then this could be your problem. "}}
{"actor":{"display_login":"MariusDieckmann"}}