{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/26561","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/26561/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/26561/comments","events_url":"https://api.github.com/repos/docker/docker/issues/26561/events","html_url":"https://github.com/docker/docker/issues/26561","id":176860018,"number":26561,"title":"deadlock in monitor.go","user":{"login":"saiwl","id":5700192,"avatar_url":"https://avatars.githubusercontent.com/u/5700192?v=3","gravatar_id":"","url":"https://api.github.com/users/saiwl","html_url":"https://github.com/saiwl","followers_url":"https://api.github.com/users/saiwl/followers","following_url":"https://api.github.com/users/saiwl/following{/other_user}","gists_url":"https://api.github.com/users/saiwl/gists{/gist_id}","starred_url":"https://api.github.com/users/saiwl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saiwl/subscriptions","organizations_url":"https://api.github.com/users/saiwl/orgs","repos_url":"https://api.github.com/users/saiwl/repos","events_url":"https://api.github.com/users/saiwl/events{/privacy}","received_events_url":"https://api.github.com/users/saiwl/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-14T09:35:13Z","updated_at":"2016-09-14T09:40:28Z","closed_at":null,"body":"<!--\r\nIf you are reporting a new issue, make sure that we do not have any duplicates\r\nalready open. You can ensure this by searching the issue list for this\r\nrepository. If there is a duplicate, please close your issue and add a comment\r\nto the existing issue instead.\r\n\r\nIf you suspect your issue is a bug, please edit your issue description to\r\ninclude the BUG REPORT INFORMATION shown below. If you fail to provide this\r\ninformation within 7 days, we cannot debug your issue and will close it. We\r\nwill, however, reopen it if you later provide the information.\r\n\r\nFor more information about reporting issues, see\r\nhttps://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues\r\n\r\n---------------------------------------------------\r\nBUG REPORT INFORMATION\r\n---------------------------------------------------\r\nUse the commands below to provide key information from your environment:\r\nYou do NOT have to include this information if this is a FEATURE REQUEST\r\n-->\r\n\r\n**Description**\r\n\r\n<!--\r\nBriefly describe the problem your having in a few paragraphs.\r\n-->\r\nDocker ps command hangs after the daemon process runs for a few days. Usually many containers has been killed and start on the host. I still cannot reproduce this problem manually. But it happens definitely after daemon runs for a few days. \r\n**Steps to reproduce the issue:**\r\n1.\r\n2.\r\n3.\r\n\r\n**Describe the results you received:**\r\n\r\n\r\n**Describe the results you expected:**\r\n\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.0\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   8eab29e\r\n Built:        Thu Jul 28 23:54:00 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.0\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   8eab29e\r\n Built:        Thu Jul 28 23:54:00 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.0\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   8eab29e\r\n Built:        Thu Jul 28 23:54:00 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.0\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   8eab29e\r\n Built:        Thu Jul 28 23:54:00 2016\r\n OS/Arch:      linux/amd64\r\nwanglu@mesos22.aws.dev:~$ sudo docker info\r\nContainers: 12045\r\n Running: 29\r\n Paused: 0\r\n Stopped: 12016\r\nImages: 50\r\nServer Version: 1.12.0\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null bridge host overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 4.5.0-0.bpo.2-amd64\r\nOperating System: Debian GNU/Linux 8 (jessie)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 8\r\nTotal Memory: 14.69 GiB\r\nName: mesos22\r\nID: WFEI:YJJD:2XQ3:GQHW:YVQZ:UCPT:NSLJ:KEPN:TDKS:WDZ7:PLSV:IZJN\r\nDocker Root Dir: /data/data/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/246959710","html_url":"https://github.com/docker/docker/issues/26561#issuecomment-246959710","issue_url":"https://api.github.com/repos/docker/docker/issues/26561","id":246959710,"user":{"login":"saiwl","id":5700192,"avatar_url":"https://avatars.githubusercontent.com/u/5700192?v=3","gravatar_id":"","url":"https://api.github.com/users/saiwl","html_url":"https://github.com/saiwl","followers_url":"https://api.github.com/users/saiwl/followers","following_url":"https://api.github.com/users/saiwl/following{/other_user}","gists_url":"https://api.github.com/users/saiwl/gists{/gist_id}","starred_url":"https://api.github.com/users/saiwl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saiwl/subscriptions","organizations_url":"https://api.github.com/users/saiwl/orgs","repos_url":"https://api.github.com/users/saiwl/repos","events_url":"https://api.github.com/users/saiwl/events{/privacy}","received_events_url":"https://api.github.com/users/saiwl/received_events","type":"User","site_admin":false},"created_at":"2016-09-14T09:40:28Z","updated_at":"2016-09-14T09:40:28Z","body":"I got the stack dump file. Here is the related part:\r\n```\r\nngoroutine 776185020 [semacquire, 1400 minutes]:\\nsync.runtime_Semacquire(0xc825372fcc)\\n\\t/usr/local/go/src/runtime/sema.go:47 +0x26\\nsync.(*WaitGroup).Wait(0xc825372fc0)\\n\\t/usr/local/go/src/sync/waitgroup.go:127 +0xb4\\ngithub.com/docker/docker/daemon.(*Daemon).StateChanged(0xc8200721a0, 0xc83bcab800, 0x40, 0x1e1cbf0, 0xc, 0x0, 0xc83bcab880, 0x40, 0x0, 0x0, ...)\\n\\t/go/src/github.com/docker/docker/daemon/monitor.go:68 +0xecc\\ngithub.com/docker/docker/libcontainerd.(*container).handleEvent.func2()\\n\\t/go/src/github.com/docker/docker/libcontainerd/container_linux.go:202 +0x9e\\ngithub.com/docker/docker/libcontainerd.(*queue).append.func1(0xc837c09c01, 0xc830b9c060, 0xc83dfcc4c0, 0xc828486480)\\n\\t/go/src/github.com/docker/docker/libcontainerd/queue_linux.go:26 +0x47\\ncreated by github.com/docker/docker/libcontainerd.(*queue).append\\n\\t/go/src/github.com/docker/docker/libcontainerd/queue_linux.go:28 +0x1da\\n\\n```\r\nThe related code shows below:\r\n\r\n```\r\ncase libcontainerd.StateExitProcess:\r\n\t\tc.Lock()\r\n\t\tdefer c.Unlock()\r\n\t\tif execConfig := c.ExecCommands.Get(e.ProcessID); execConfig != nil {\r\n\t\t\tec := int(e.ExitCode)\r\n\t\t\texecConfig.ExitCode = &ec\r\n\t\t\texecConfig.Running = false\r\n\t\t\texecConfig.Wait()\r\n\t\t\tif err := execConfig.CloseStreams(); err != nil {\r\n\t\t\t\tlogrus.Errorf(\"%s: %s\", c.ID, err)\r\n\t\t\t}\r\n\r\n\t\t\t// remove the exec command from the container's store only and not the\r\n\t\t\t// daemon's store so that the exec command can be inspected.\r\n\t\t\tc.ExecCommands.Delete(execConfig.ID)\r\n\t\t} else {\r\n\t\t\tlogrus.Warnf(\"Ignoring StateExitProcess for %v but no exec command found\", e)\r\n\t\t}\r\n```\r\nIt stucks at the execConfig.Wait()"}}
{"actor":{"display_login":"saiwl"}}