{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/weave/issues/2559","repository_url":"https://api.github.com/repos/weaveworks/weave","labels_url":"https://api.github.com/repos/weaveworks/weave/issues/2559/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/weave/issues/2559/comments","events_url":"https://api.github.com/repos/weaveworks/weave/issues/2559/events","html_url":"https://github.com/weaveworks/weave/issues/2559","id":185149158,"number":2559,"title":"weave attach --rewrite-hosts sporadically fails","user":{"login":"SpComb","id":3950373,"avatar_url":"https://avatars.githubusercontent.com/u/3950373?v=3","gravatar_id":"","url":"https://api.github.com/users/SpComb","html_url":"https://github.com/SpComb","followers_url":"https://api.github.com/users/SpComb/followers","following_url":"https://api.github.com/users/SpComb/following{/other_user}","gists_url":"https://api.github.com/users/SpComb/gists{/gist_id}","starred_url":"https://api.github.com/users/SpComb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SpComb/subscriptions","organizations_url":"https://api.github.com/users/SpComb/orgs","repos_url":"https://api.github.com/users/SpComb/repos","events_url":"https://api.github.com/users/SpComb/events{/privacy}","received_events_url":"https://api.github.com/users/SpComb/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/weaveworks/weave/labels/state/investigating","name":"state/investigating","color":"b60205"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-25T15:19:57Z","updated_at":"2016-10-27T08:32:39Z","closed_at":null,"body":"There seems to be some race condition with the the implementation of `weave attach --rewrite-hosts` in the following [system configuration](https://gist.github.com/SpComb/f56d21c75507a13c22889eea7611ba21#file-docker-info):\r\n\r\n* Ubuntu xenial Linux 4.4\r\n* Weave 1.7.2\r\n* Docker 1.12 (both Ubuntu `docker.io=1.12.1-0ubuntu13~16.04.1` and upstream `docker-engine=1.12.2-0~xenial`)\r\n * Storage Driver: overlay\r\n\r\n\r\nThe `weave attach --rewrite-hosts` operation will sometimes succeed, and sometimes fail with an error:\r\n\r\n```\r\nError response from daemon: Driver overlay failed to remove root filesystem cc8482b244b5066b1d42f3a4e7a4352abba5bb79f431a2620806b993934f8938: remove /var/lib/docker/overlay/ea588807154a1a3b04c5ff3db248f9055695fca5fe21df8341e40b11104fff90/merged/sbin: directory not empty\r\n```\r\n\r\nThis causes the container to remain detached from weave (no `ethwe` interface), and leaves large numbers of dead weaveexec containers in [`docker ps -a`](https://gist.github.com/SpComb/f56d21c75507a13c22889eea7611ba21#file-docker-ps).\r\n\r\nThis happens fairly consistently for me on a three-node cluster of DigitalIOcean nodes using their ubuntu-16-04-x64 image, leaving the majority of containers detached.\r\n\r\nMore detailed [debugging](https://gist.github.com/SpComb/f56d21c75507a13c22889eea7611ba21#file-docker-logs-a2213cd16023-L226) shows that this errors happens in `rewrite_etc_hosts()`:\r\n\r\n```\r\n+ docker run --rm --net=none -e WEAVE_CIDR=none --privileged -v /var/lib/docker/containers/2b39aafa954c370e4f950cac2ed4114197809d77c254570bcebf10281736dc05:/container --entrypoint=sh weaveworks/weaveexec:1.7.2 -c echo '# created by Weave - BEGIN ... > /container/hosts && rm -f /container/hosts && echo '# created by Weave - BEGIN ... END' > /container/hosts\r\ntime=\"2016-10-25T13:46:50Z\" level=fatal msg=\"Error response from daemon: Driver overlay failed to remove root filesystem e58b019d3e9bd44580e01607422cff527275621243046e25263844117ef90522: remove /var/lib/docker/overlay/96cf34987cc5b1c9d07de939c33726d7364f4e0927a5d9a51083349b5e81f02b/merged/sbin: directory not empty\" \r\n```\r\n\r\nIt's not exactly the same error, but this may be related to sharing mounts between namespaces: https://github.com/docker/docker/issues/26372\r\n\r\n>  This error is returned from the kernel when trying to remove container rw layers and/or the container config root... typically due to mounts leaked into other namespaces."},"comment":{"url":"https://api.github.com/repos/weaveworks/weave/issues/comments/256580411","html_url":"https://github.com/weaveworks/weave/issues/2559#issuecomment-256580411","issue_url":"https://api.github.com/repos/weaveworks/weave/issues/2559","id":256580411,"user":{"login":"awh","id":1504438,"avatar_url":"https://avatars.githubusercontent.com/u/1504438?v=3","gravatar_id":"","url":"https://api.github.com/users/awh","html_url":"https://github.com/awh","followers_url":"https://api.github.com/users/awh/followers","following_url":"https://api.github.com/users/awh/following{/other_user}","gists_url":"https://api.github.com/users/awh/gists{/gist_id}","starred_url":"https://api.github.com/users/awh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awh/subscriptions","organizations_url":"https://api.github.com/users/awh/orgs","repos_url":"https://api.github.com/users/awh/repos","events_url":"https://api.github.com/users/awh/events{/privacy}","received_events_url":"https://api.github.com/users/awh/received_events","type":"User","site_admin":false},"created_at":"2016-10-27T08:32:39Z","updated_at":"2016-10-27T08:32:39Z","body":"Original report via Slack: https://weaveworks.slack.com/archives/weave-users/p1477404926002570"}}
{"actor":{"display_login":"awh"}}