{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/27185","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/27185/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/27185/comments","events_url":"https://api.github.com/repos/docker/docker/issues/27185/events","html_url":"https://github.com/docker/docker/issues/27185","id":181367350,"number":27185,"title":"Docker Engine hangs after calling \"docker stop\"","user":{"login":"foxylion","id":2699280,"avatar_url":"https://avatars.githubusercontent.com/u/2699280?v=3","gravatar_id":"","url":"https://api.github.com/users/foxylion","html_url":"https://github.com/foxylion","followers_url":"https://api.github.com/users/foxylion/followers","following_url":"https://api.github.com/users/foxylion/following{/other_user}","gists_url":"https://api.github.com/users/foxylion/gists{/gist_id}","starred_url":"https://api.github.com/users/foxylion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/foxylion/subscriptions","organizations_url":"https://api.github.com/users/foxylion/orgs","repos_url":"https://api.github.com/users/foxylion/repos","events_url":"https://api.github.com/users/foxylion/events{/privacy}","received_events_url":"https://api.github.com/users/foxylion/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/status/more-info-needed","name":"status/more-info-needed","color":"C6A49A"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-06T09:38:09Z","updated_at":"2016-10-10T15:02:39Z","closed_at":null,"body":"**Description**\r\nThis issue is sporadic and does only occur once a day on our Jenkins node. We are running our Jobs in a Docker container using the Jenkins Docker pipeline plugin. At the end of a Job the plugin stops the container and will hang from there on.\r\n\r\n**Steps to reproduce the issue:**\r\n1. This is sporadic and not reproducible step by step\r\n\r\n**Describe the results you received:**\r\nNo repsonse from daemon to `docker ps`, `docker stop <container>`, `docker kill <container>`\r\n\r\n**Describe the results you expected:**\r\nNormal responses when calling `docker ps`, ...\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        Thu Aug 18 05:22:43 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        Thu Aug 18 05:22:43 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 5\r\n Running: 1\r\n Paused: 0\r\n Stopped: 4\r\nImages: 2\r\nServer Version: 1.12.1\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 29\r\n Dirperm1 Supported: false\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: host bridge null overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: apparmor\r\nKernel Version: 3.13.0-48-generic\r\nOperating System: Ubuntu 14.04.4 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 1\r\nTotal Memory: 992.5 MiB\r\nName: jenkins-node-1.internal\r\nID: IOJZ:PGC4:4HVY:4DKY:32EP:...\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 23\r\n Goroutines: 45\r\n System Time: 2016-10-06T09:33:37.423801303Z\r\n EventsListeners: 0\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No swap limit support\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nDocker is running on an EC2 t2.micro instance with Ubuntu 14.04.3.\r\n\r\nLog output of a normal \"docker stop\" called from the Jenkins plugin:\r\n```\r\ntime=\"2016-10-05T13:59:28.707836999Z\" level=debug msg=\"Calling POST /v1.24/containers/75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a/stop?t=10\" \r\ntime=\"2016-10-05T13:59:28.707907143Z\" level=debug msg=\"Sending 15 to 75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a\" \r\ntime=\"2016-10-05T13:59:38.708608718Z\" level=info msg=\"Container 75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2016-10-05T13:59:38.708649191Z\" level=debug msg=\"Sending 9 to 75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a\" ESC[37mDEBUESC[0m[5863] containerd: process exited                    ESC[37midESC[0m=75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a ESC[37mpidESC[0m=init ESC[37mstatusESC[0m=137 ESC[37msystemPidESC[0m=9973\r\ntime=\"2016-10-05T13:59:38.715182786Z\" level=debug msg=\"libcontainerd: received containerd event: &types.Event{Type:\\\"exit\\\", Id:\\\"75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a\\\", Status:0x89, Pid:\\\"init\\\", Timestamp:(*timestamp.Timestamp)(0xc820772e00)}\" \r\ntime=\"2016-10-05T13:59:38.715501347Z\" level=debug msg=\"Revoking external connectivity on endpoint pedantic_franklin (57dd9c924af5b9c48f9d193b748aa57dc8efb101786132ecdc1c55842979ec8c)\" \r\ntime=\"2016-10-05T13:59:38.756258676Z\" level=debug msg=\"Releasing addresses for endpoint pedantic_franklin's interface on network bridge\" \r\ntime=\"2016-10-05T13:59:38.756291308Z\" level=debug msg=\"ReleaseAddress(LocalDefault/10.42.0.0/16, 10.42.0.2)\" \r\ntime=\"2016-10-05T13:59:39.938467748Z\" level=debug msg=\"Calling DELETE /v1.24/containers/75190c7ed862395af5a4dfed025f6f27a3bb39da9393a61ef724e698ab8d325a?force=1\" \r\n\r\ntime=\"2016-10-05T14:01:56.270713926Z\" level=debug msg=\"clean 182 unused exec commands\" \r\ntime=\"2016-10-05T14:06:56.270678517Z\" level=debug msg=\"clean 152 unused exec commands\" \r\n```\r\n\r\nLog output of a \"docker stop\" called from the Jenkins plugin when the daemon does not respond afterwards:\r\n```\r\ntime=\"2016-10-05T14:59:33.611998832Z\" level=debug msg=\"Calling POST /v1.24/containers/b78f9f03b798f25a9eee84ac9db53f20bb7a6edf8059b640846c128b9f1f1e59/stop?t=10\" \r\n<< nothing, after calling stop, follows >>\r\n```\r\n\r\ndmesg and docker.log after calling `kill -s SIGUSR1 <PID>`:\r\nhttps://gist.github.com/foxylion/166ffed66c95e46a7e12dd5c54447794\r\n\r\nAfter such a container hangs it is possible to call `docker run`, but not `docker ps` or `docker kill b78f9f03b798f25a9eee84ac9db53f20bb7a6edf8059b640846c128b9f1f1e59` to get rid of the hanging container."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/252649318","html_url":"https://github.com/docker/docker/issues/27185#issuecomment-252649318","issue_url":"https://api.github.com/repos/docker/docker/issues/27185","id":252649318,"user":{"login":"cpuguy83","id":799078,"avatar_url":"https://avatars.githubusercontent.com/u/799078?v=3","gravatar_id":"","url":"https://api.github.com/users/cpuguy83","html_url":"https://github.com/cpuguy83","followers_url":"https://api.github.com/users/cpuguy83/followers","following_url":"https://api.github.com/users/cpuguy83/following{/other_user}","gists_url":"https://api.github.com/users/cpuguy83/gists{/gist_id}","starred_url":"https://api.github.com/users/cpuguy83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpuguy83/subscriptions","organizations_url":"https://api.github.com/users/cpuguy83/orgs","repos_url":"https://api.github.com/users/cpuguy83/repos","events_url":"https://api.github.com/users/cpuguy83/events{/privacy}","received_events_url":"https://api.github.com/users/cpuguy83/received_events","type":"User","site_admin":false},"created_at":"2016-10-10T15:02:39Z","updated_at":"2016-10-10T15:02:39Z","body":"@mlaventure I'm not sure since I've only glanced on this, but looks like it's stuck waiting for an exec to finish here: https://gist.github.com/foxylion/166ffed66c95e46a7e12dd5c54447794#file-docker-log-L715"}}
{"actor":{"display_login":"cpuguy83"}}