{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/37663","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/37663/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/37663/comments","events_url":"https://api.github.com/repos/moby/moby/issues/37663/events","html_url":"https://github.com/moby/moby/issues/37663","id":351692605,"node_id":"MDU6SXNzdWUzNTE2OTI2MDU=","number":37663,"title":"Containers take take a very long time to stop after all processes in them have exited","user":{"login":"orodbhen","id":4573865,"node_id":"MDQ6VXNlcjQ1NzM4NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/4573865?v=4","gravatar_id":"","url":"https://api.github.com/users/orodbhen","html_url":"https://github.com/orodbhen","followers_url":"https://api.github.com/users/orodbhen/followers","following_url":"https://api.github.com/users/orodbhen/following{/other_user}","gists_url":"https://api.github.com/users/orodbhen/gists{/gist_id}","starred_url":"https://api.github.com/users/orodbhen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orodbhen/subscriptions","organizations_url":"https://api.github.com/users/orodbhen/orgs","repos_url":"https://api.github.com/users/orodbhen/repos","events_url":"https://api.github.com/users/orodbhen/events{/privacy}","received_events_url":"https://api.github.com/users/orodbhen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-17T18:18:43Z","updated_at":"2018-08-24T12:39:56Z","closed_at":null,"author_association":"NONE","body":"This may be related to #34097, as the symptoms are very similar to those reported. However, I was instructed to open a new ticket. \r\n\r\nSummary:\r\n\r\n- Run `docker stop -t 6`\r\n- Logs confirm that SIGTERM is sent to the container\r\n- The `top` command shows that all processes in the container exit almost immediately after they receive the SIGTERM.\r\n- The `docker ps` command shows the container is still running long after all processes have exited. \r\n- From the daemon logs, an attempt is made to force the container down after the 6 second timeout expires. \r\n- The container takes as much as 1.5 minutes to stop, after the daemon tries to force it down. \r\n- Changing the timeout (e.g. 20 sec) makes not difference. \r\n- Using `docker kill -s TERM` followed by `docker kill` (after timeout) behaves the same as `docker stop -t <timeout>`. \r\n\r\nThe main process in the container spawns all the other processes, and waits for a terminate or interrupt signal. When it's received, it terminates all child processes and exits. Running the `top` command on the host, and filtering the output for the `uid` used in the container, shows all container processes quickly exiting after the `stop` command is run. The container logs also confirm that the shutdown process occurred as expected. \r\n\r\nThere are between 2-15 containers running in a group. When only 3 or 4 containers are run, they shutdown as expected. CPU and memory usage are high with all 15 instances running, but not higher than expected and well below the maximum supported by hardware. \r\n\r\nI restarted the daemon in debug mode. After 6 seconds the logs show:\r\n\r\n> failed to exit within 6 seconds - using the force\r\n\r\nThe same message occurs for each container in the group. \r\n\r\nImmediately after it prints for a single container:\r\n\r\n> kill failed because 'container not found' or 'no such process'\r\n\r\nAbout a 1:20 minutes later:\r\n\r\n> Container <id> is not running\r\n\r\nThis message was for a different container ID than the previous one. \r\n\r\nNo other warning or error messages.\r\n\r\nThis behavior is very consistent, and sometimes results in other non-docker terminal commands taking a very long time to execute while it's occurring. \r\n\r\n**Platform:** \r\n\r\n- CentOS 7.5, on a single bare-metal server (no swarm)\r\n- Server version: 18.03.1-ce\r\n- Kernel version: 3.10.0-862.3.3.el7.x86_64\r\n- Storage driver: overlay\r\n- Backing filesystem: xfs\r\n\r\nI tried this on a system running `17.04` as well, with the same results. \r\n\r\n**Note:** This is an air-gap system, so I'm not able to copy and paste logs or command output. \r\n\r\nA couple question I have are:\r\n\r\n- What exactly does Docker do when stopping a container?\r\n- What constitutes the container being \"stopped\"?\r\n"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/415746716","html_url":"https://github.com/moby/moby/issues/37663#issuecomment-415746716","issue_url":"https://api.github.com/repos/moby/moby/issues/37663","id":415746716,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTc0NjcxNg==","user":{"login":"orodbhen","id":4573865,"node_id":"MDQ6VXNlcjQ1NzM4NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/4573865?v=4","gravatar_id":"","url":"https://api.github.com/users/orodbhen","html_url":"https://github.com/orodbhen","followers_url":"https://api.github.com/users/orodbhen/followers","following_url":"https://api.github.com/users/orodbhen/following{/other_user}","gists_url":"https://api.github.com/users/orodbhen/gists{/gist_id}","starred_url":"https://api.github.com/users/orodbhen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orodbhen/subscriptions","organizations_url":"https://api.github.com/users/orodbhen/orgs","repos_url":"https://api.github.com/users/orodbhen/repos","events_url":"https://api.github.com/users/orodbhen/events{/privacy}","received_events_url":"https://api.github.com/users/orodbhen/received_events","type":"User","site_admin":false},"created_at":"2018-08-24T12:39:56Z","updated_at":"2018-08-24T12:39:56Z","author_association":"NONE","body":"It looks like this was being caused by a change to the software that was running in the container. It was writing log files to `/tmp` which was not mounted to an external volume. Mounting it to a `tmpfs` volume fixed the problem. \r\n\r\nHowever, I'm hesitant to say that this issue should be closed, as the behavior I was seeing doesn't seem appropriate for the cause. Why should a container hang for over a minute after it's processes have stopped, as a result of this?\r\n\r\nIt also appeared to be the cause of other sporadic errors I was getting from Docker and docker-compose, that had no obvious connection. Examples:\r\n\r\n- `UnixHTTPConnectionPool` exceptions.\r\n- The composition taking a long time to start, failing, and then succeeding on the next try. \r\n- et alia.\r\n\r\n Sometimes a restart of the Daemon was required to get things working again. I'll wait to see if these errors reoccur before posting more details, but they appear to have stopped. \r\n"}}
{"actor":{"display_login":"orodbhen"}}