{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/19030","labels_url":"https://api.github.com/repos/docker/docker/issues/19030/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/19030/comments","events_url":"https://api.github.com/repos/docker/docker/issues/19030/events","html_url":"https://github.com/docker/docker/issues/19030","id":124580386,"number":19030,"title":"ISSUE: Can't stop containers (sometimes but often enough to test)","user":{"login":"beetree","id":12211793,"avatar_url":"https://avatars.githubusercontent.com/u/12211793?v=3","gravatar_id":"","url":"https://api.github.com/users/beetree","html_url":"https://github.com/beetree","followers_url":"https://api.github.com/users/beetree/followers","following_url":"https://api.github.com/users/beetree/following{/other_user}","gists_url":"https://api.github.com/users/beetree/gists{/gist_id}","starred_url":"https://api.github.com/users/beetree/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beetree/subscriptions","organizations_url":"https://api.github.com/users/beetree/orgs","repos_url":"https://api.github.com/users/beetree/repos","events_url":"https://api.github.com/users/beetree/events{/privacy}","received_events_url":"https://api.github.com/users/beetree/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-01-02T07:54:35Z","updated_at":"2016-01-04T04:43:10Z","closed_at":null,"body":"I run a cluster where docker containers *sometimes* become \"unstoppable\". It happens a few times per day across the cluster, and when it happens the only solution is to stop the docker daemon, restart the host machine and then start the docker daemon again.\r\n\r\nA regular stop command does not do the job (even when waiting for up to an hour):\r\n```\r\n# time docker stop --time=1 950677e2317f\r\n^C\r\nreal    0m13.508s\r\nuser    0m0.036s\r\nsys     0m0.008s\r\n```\r\n\r\nThe daemon seems to properly escalate the stop of the container:\r\n```\r\n# journalctl -fu docker.service\r\n-- Logs begin at Fri 2015-12-11 15:40:55 CET. --\r\nDec 31 23:30:33 m3561.contabo.host docker[9988]: time=\"2015-12-31T23:30:33.164731953+01:00\" level=info msg=\"POST /v1.21/containers/950677e2317f/stop?t=1\"\r\nDec 31 23:30:34 m3561.contabo.host docker[9988]: time=\"2015-12-31T23:30:34.165531990+01:00\" level=info msg=\"Container 950677e2317fcd2403ef5b5ffad37204e880136e91f76b0a8682e04a93e80942 failed to exit within 1 seconds of SIGTERM - using the force\"\r\nDec 31 23:30:44 m3561.contabo.host docker[9988]: time=\"2015-12-31T23:30:44.165954266+01:00\"\r\n```\r\n\r\nThe process that seems to be blocking the container stop can be seen on the host machine:\r\n```\r\n# ps aux | grep [1]1991\r\nroot     11991 84.3  0.0   5836   132 ?        R    Dec30 1300:19 bash -c (echo stop > /tmp/minecraft &)\r\n# top -b | grep [1]1991\r\n11991 root      20   0    5836    132     20 R  89.5  0.0   1300:29 bash\r\n```\r\n\r\nPlease note that it is not a [Z]ombie process but rather a [R]unning process. Also note that it's consuming 84% of the cpu so it is actively doing *something* and just leaving it is not an option.\r\n\r\nSome base information about my setup:\r\n```\r\n# docker version\r\nClient:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:20:08 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:20:08 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\n# docker info\r\nContainers: 189\r\nImages: 322\r\nServer Version: 1.9.1\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 700\r\n Dirperm1 Supported: true\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 4.2.0-19-generic\r\nOperating System: Ubuntu 15.10\r\nCPUs: 24\r\nTotal Memory: 125.8 GiB\r\nName: m3561.contabo.host\r\nID: ZM2Q:RA6Q:E4NM:5Q2Q:R7E4:BFPQ:EEVK:7MEO:YRH6:SVS6:RIHA:3I2K\r\n\r\n# uname -a\r\nLinux m3561.contabo.host 4.2.0-19-generic #23-Ubuntu SMP Wed Nov 11 11:39:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\nWorth noting is that stopping the docker daemon does not solve the issue as the process is still running also after the daemon is stopped. This may indicate that the issue is not related to Docker but rather to the kernel.\r\n\r\nI've seen 20+ of these \"stuck processes\" and the only \"theme\" I have noticed is that it seems that they always involve some form of piping of data. Not sure if this is helpful in trying to understand what's going on here.\r\n\r\nHappy to test anything to resolve this. `kill -9` on the host machine doesn't solve it. I can't run commands from within the container using `docker exec` so can't try killing the process from within.I can get plenty of information of the process from `/proc/`, just say what information you want and I'll get it. Unfortunately, I can't reproduce the issue (yet) but I regularly (every day) catch it in my live environment. Updating to a different docker version to \"test if it solves it\" is somewhat costly as it would involve updating the whole cluster, but if there is a good hypothesis that it will solve the issue I'll go through the work to do it, else my plan is to wait for next stable release until updating.\r\n\r\nReally appreciate any help on this as the issue is severely lowering the reliability of the cluster as it requires several-per-day restarts of entire Nodes. Big thanks!\r\n\r\n( [This stackoverflow post](https://stackoverflow.com/questions/34552232/cant-kill-processes-originating-in-a-docker-container) is a duplicate. )\r\n\r\nUPDATE: (adding examples of crashed processes)\r\n```\r\n# ps aux | grep [1]8342\r\nroot     18342 92.0  0.0   5836   132 ?        R     2015 2279:07 bash -c (echo stop > /tmp/minecraft &)\r\n```\r\n\r\n```\r\n# ps aux | grep [3]1572\r\nroot     31572 95.5  0.0   4448   104 ?        R     2015 2946:35 /bin/sh -c (redis-server &>redis.log &) && ./setup-wait.sh && sleep 3 && ./nodebb start && ./nodebb log && sleep infinity\r\n```\r\n\r\n/beetree\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/168580491","html_url":"https://github.com/docker/docker/issues/19030#issuecomment-168580491","issue_url":"https://api.github.com/repos/docker/docker/issues/19030","id":168580491,"user":{"login":"AkihiroSuda","id":9248427,"avatar_url":"https://avatars.githubusercontent.com/u/9248427?v=3","gravatar_id":"","url":"https://api.github.com/users/AkihiroSuda","html_url":"https://github.com/AkihiroSuda","followers_url":"https://api.github.com/users/AkihiroSuda/followers","following_url":"https://api.github.com/users/AkihiroSuda/following{/other_user}","gists_url":"https://api.github.com/users/AkihiroSuda/gists{/gist_id}","starred_url":"https://api.github.com/users/AkihiroSuda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AkihiroSuda/subscriptions","organizations_url":"https://api.github.com/users/AkihiroSuda/orgs","repos_url":"https://api.github.com/users/AkihiroSuda/repos","events_url":"https://api.github.com/users/AkihiroSuda/events{/privacy}","received_events_url":"https://api.github.com/users/AkihiroSuda/received_events","type":"User","site_admin":false},"created_at":"2016-01-04T04:43:10Z","updated_at":"2016-01-04T04:43:10Z","body":"@beetree Sorry, I misreaded about [Z]ombie processes.\r\n"}}