{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/18758","labels_url":"https://api.github.com/repos/docker/docker/issues/18758/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/18758/comments","events_url":"https://api.github.com/repos/docker/docker/issues/18758/events","html_url":"https://github.com/docker/docker/issues/18758","id":122848136,"number":18758,"title":"hang on `docker stop`","user":{"login":"andybalholm","id":868908,"avatar_url":"https://avatars.githubusercontent.com/u/868908?v=3","gravatar_id":"","url":"https://api.github.com/users/andybalholm","html_url":"https://github.com/andybalholm","followers_url":"https://api.github.com/users/andybalholm/followers","following_url":"https://api.github.com/users/andybalholm/following{/other_user}","gists_url":"https://api.github.com/users/andybalholm/gists{/gist_id}","starred_url":"https://api.github.com/users/andybalholm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andybalholm/subscriptions","organizations_url":"https://api.github.com/users/andybalholm/orgs","repos_url":"https://api.github.com/users/andybalholm/repos","events_url":"https://api.github.com/users/andybalholm/events{/privacy}","received_events_url":"https://api.github.com/users/andybalholm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/area/kernel","name":"area/kernel","color":"E3ACEF"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-12-17T23:51:19Z","updated_at":"2016-01-27T05:35:43Z","closed_at":null,"body":"When we stop our Discourse Docker container with `docker stop app` or `service docker restart`, it hangs. One or more `runsv` processes inside the container hang, consuming 100% CPU. top shows the CPU usage as \"system\" usage, so they are hung inside the kernel somewhere. Because of this, they can't be killed even with SIGKILL; we need to reboot the server.\r\n\r\nThis is a regression that showed up after we did some routine updates on the server (the ones that Ubuntu nags you to do). The packages that were upgraded included the kernel and Docker.\r\n\r\n```\r\nroot@mail:~# uname -a\r\nLinux mail.daystar.io 3.13.0-73-generic #116-Ubuntu SMP Fri Dec 4 15:31:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\nroot@mail:~# docker version\r\nClient:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.3\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 17:56:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.3\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 17:56:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\nroot@mail:~# docker info\r\nContainers: 34\r\nImages: 131\r\nServer Version: 1.9.1\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 230\r\n Dirperm1 Supported: false\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.13.0-73-generic\r\nOperating System: Ubuntu 14.04.3 LTS\r\nCPUs: 12\r\nTotal Memory: 51.08 GiB\r\nName: mail.daystar.io\r\nID: 5PHA:QONL:U6TH:2IPO:AVNK:LMCW:R6XT:AVZZ:KNGC:T4TF:SX5S:5DHN\r\nWARNING: No swap limit support\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/175411001","html_url":"https://github.com/docker/docker/issues/18758#issuecomment-175411001","issue_url":"https://api.github.com/repos/docker/docker/issues/18758","id":175411001,"user":{"login":"SamSaffron","id":5213,"avatar_url":"https://avatars.githubusercontent.com/u/5213?v=3","gravatar_id":"","url":"https://api.github.com/users/SamSaffron","html_url":"https://github.com/SamSaffron","followers_url":"https://api.github.com/users/SamSaffron/followers","following_url":"https://api.github.com/users/SamSaffron/following{/other_user}","gists_url":"https://api.github.com/users/SamSaffron/gists{/gist_id}","starred_url":"https://api.github.com/users/SamSaffron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SamSaffron/subscriptions","organizations_url":"https://api.github.com/users/SamSaffron/orgs","repos_url":"https://api.github.com/users/SamSaffron/repos","events_url":"https://api.github.com/users/SamSaffron/events{/privacy}","received_events_url":"https://api.github.com/users/SamSaffron/received_events","type":"User","site_admin":false},"created_at":"2016-01-27T05:35:43Z","updated_at":"2016-01-27T05:35:43Z","body":"@thaJeztah @jfrazelle having just worked around this issue I think docker can (and should) workaround this problem. \r\n\r\nThe gist of what happens is: \r\n\r\n- You have a container with N processes\r\n- `docker stop` is issued\r\n- pid 1 gets a TERM \r\n- pid 1 exists but other pids are still running some may have got TERMs\r\n- In some weird kernel linux container condition the orphan processes are not torn down and trigger a failure mode in the kernel.\r\n\r\nInstead I thing the \"semantics\" of `docker stop` should be changed to:\r\n\r\n1. Send TERM to pid 1, wait up to 10 seconds for pid 1 to finish. (if not finished send a QUIT)  \r\n2. Check if any other pids are still running in the container, if there are \r\n   a. In parallel send TERM + wait for up to 5 seconds for each of the remaining pids. If it is still running send a QUIT and wait. \r\n3. Stop container\r\n\r\nThis will change the semantics a bit on docker stop, but, on the other hand, introduce some sort of documented behavior for orphans and make `stop` much more reliable even if kernel or aufs has bugs. "}}