{"action":"closed","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/5069","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/5069/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/5069/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/5069/events","html_url":"https://github.com/rancher/rancher/issues/5069","id":159637009,"number":5069,"title":"Host disconnects","user":{"login":"qrpike","id":669468,"avatar_url":"https://avatars.githubusercontent.com/u/669468?v=3","gravatar_id":"","url":"https://api.github.com/users/qrpike","html_url":"https://github.com/qrpike","followers_url":"https://api.github.com/users/qrpike/followers","following_url":"https://api.github.com/users/qrpike/following{/other_user}","gists_url":"https://api.github.com/users/qrpike/gists{/gist_id}","starred_url":"https://api.github.com/users/qrpike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qrpike/subscriptions","organizations_url":"https://api.github.com/users/qrpike/orgs","repos_url":"https://api.github.com/users/qrpike/repos","events_url":"https://api.github.com/users/qrpike/events{/privacy}","received_events_url":"https://api.github.com/users/qrpike/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"ibuildthecloud","id":1754002,"avatar_url":"https://avatars.githubusercontent.com/u/1754002?v=3","gravatar_id":"","url":"https://api.github.com/users/ibuildthecloud","html_url":"https://github.com/ibuildthecloud","followers_url":"https://api.github.com/users/ibuildthecloud/followers","following_url":"https://api.github.com/users/ibuildthecloud/following{/other_user}","gists_url":"https://api.github.com/users/ibuildthecloud/gists{/gist_id}","starred_url":"https://api.github.com/users/ibuildthecloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibuildthecloud/subscriptions","organizations_url":"https://api.github.com/users/ibuildthecloud/orgs","repos_url":"https://api.github.com/users/ibuildthecloud/repos","events_url":"https://api.github.com/users/ibuildthecloud/events{/privacy}","received_events_url":"https://api.github.com/users/ibuildthecloud/received_events","type":"User","site_admin":false},"assignees":[{"login":"ibuildthecloud","id":1754002,"avatar_url":"https://avatars.githubusercontent.com/u/1754002?v=3","gravatar_id":"","url":"https://api.github.com/users/ibuildthecloud","html_url":"https://github.com/ibuildthecloud","followers_url":"https://api.github.com/users/ibuildthecloud/followers","following_url":"https://api.github.com/users/ibuildthecloud/following{/other_user}","gists_url":"https://api.github.com/users/ibuildthecloud/gists{/gist_id}","starred_url":"https://api.github.com/users/ibuildthecloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibuildthecloud/subscriptions","organizations_url":"https://api.github.com/users/ibuildthecloud/orgs","repos_url":"https://api.github.com/users/ibuildthecloud/repos","events_url":"https://api.github.com/users/ibuildthecloud/events{/privacy}","received_events_url":"https://api.github.com/users/ibuildthecloud/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/67","html_url":"https://github.com/rancher/rancher/milestones/Release%201.2.0","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/67/labels","id":1788744,"number":67,"title":"Release 1.2.0","description":"","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":286,"closed_issues":120,"state":"open","created_at":"2016-05-26T00:51:12Z","updated_at":"2016-10-15T02:56:23Z","due_on":"2016-10-28T07:00:00Z","closed_at":null},"comments":2,"created_at":"2016-06-10T13:38:23Z","updated_at":"2016-10-15T02:56:23Z","closed_at":"2016-10-15T02:56:23Z","body":"**Rancher Version:**\r\n`1.1.0-dev3` - Also worrying that rancher/server:latest upgrades to dev instead of a stable branch. \r\n\r\n**Docker Version:**\r\n1.10.3\r\n\r\n**OS and where are the hosts located? (cloud, bare metal, etc):**\r\n`CoreOS 1010.4` 4gb box on DigitalOcean, slaves are 16gb boxes.\r\n\r\n**Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)**\r\nSingle node master with multiple slaves. Internal DB mounted on host. \r\n\r\n**Environment Type: (Cattle/Kubernetes/Swarm/Mesos)**\r\nCattle\r\n\r\n**Steps to Reproduce:**\r\nWe have been running rancher in all our environments for a while now and all the sudden we are getting random disconnects, that do not automatically reconnect in this environment. I have to ssh in and restart the `rancher-agent` container. Then everything is back to normal for about a day until it happens again. It only happens in the Digital Ocean environment. It seems the disk space command is failing quite often.\r\n\r\n\r\n\r\n```\r\ne04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/upper/var/lib/cattle/download.x02d2b2': No such file or directory\r\n - exit status 1\r\nE0610 11:06:17.816614    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 660424\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/upper/var/lib/cattle/download.SG9hTIm': No such file or directory\r\n - exit status 1\r\nE0610 11:10:21.934690    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 660504\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/upper/var/lib/cattle/download.tRTy8Z2': No such file or directory\r\n - exit status 1\r\nE0610 11:39:57.026114    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 660960\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.ft94zEO': No such file or directory\r\n - exit status 1\r\nE0610 11:57:19.948862    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 661324\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.nLCFeqM': No such file or directory\r\n - exit status 1\r\nE0610 12:08:36.040981    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 661568\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.EZYlhpr': No such file or directory\r\n - exit status 1\r\nE0610 12:12:40.929771    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 661476\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.k3giG2n': No such file or directory\r\n - exit status 1\r\nE0610 12:17:44.778339    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 661952\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/upper/var/lib/cattle/download.F36rvTp': No such file or directory\r\n - exit status 1\r\nE0610 12:20:48.356610    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 661828\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.4BANHfX': No such file or directory\r\n - exit status 1\r\nE0610 12:25:54.403978    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 662128\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.PsBrG4q': No such file or directory\r\ndu: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/upper/var/lib/cattle/download.8N4gxPH': No such file or directory\r\n - exit status 1\r\nE0610 12:40:10.972677    7003 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016 with output stdout: 662368\t/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016\r\n, stderr: du: cannot access '/var/lib/docker/overlay/0a96a78ee5a56e8dd743b98148775e04e6a604c91f466f0c0af8d9f3998a9016/merged/var/lib/cattle/download.OLryWEJ': No such file or directory\r\n - exit status 1\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\nINFO: Starting agent for F491E98FB1014BA7B279\r\n```\r\n\r\n\r\n```\r\n$df -h\r\nFilesystem      Size  Used Avail Use% Mounted on\r\ndevtmpfs        2.0G     0  2.0G   0% /dev\r\ntmpfs           2.0G     0  2.0G   0% /dev/shm\r\ntmpfs           2.0G  1.3M  2.0G   1% /run\r\ntmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup\r\n/dev/vda9        56G   23G   32G  43% /\r\n/dev/vda3       985M  616M  318M  66% /usr\r\n/dev/vda1       128M   67M   61M  53% /boot\r\ntmpfs           2.0G     0  2.0G   0% /media\r\ntmpfs           2.0G     0  2.0G   0% /tmp\r\n/dev/vda6       108M   60K   99M   1% /usr/share/oem\r\n```"}}
{"actor":{"display_login":"deniseschannon"}}