{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/36661","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/36661/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/36661/comments","events_url":"https://api.github.com/repos/moby/moby/issues/36661/events","html_url":"https://github.com/moby/moby/issues/36661","id":307418654,"number":36661,"title":"Container HEALTHCHECKs can lead to hanging API calls and bad State","user":{"login":"jahkeup","id":1139752,"avatar_url":"https://avatars2.githubusercontent.com/u/1139752?v=4","gravatar_id":"","url":"https://api.github.com/users/jahkeup","html_url":"https://github.com/jahkeup","followers_url":"https://api.github.com/users/jahkeup/followers","following_url":"https://api.github.com/users/jahkeup/following{/other_user}","gists_url":"https://api.github.com/users/jahkeup/gists{/gist_id}","starred_url":"https://api.github.com/users/jahkeup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jahkeup/subscriptions","organizations_url":"https://api.github.com/users/jahkeup/orgs","repos_url":"https://api.github.com/users/jahkeup/repos","events_url":"https://api.github.com/users/jahkeup/events{/privacy}","received_events_url":"https://api.github.com/users/jahkeup/received_events","type":"User","site_admin":false},"labels":[{"id":73641990,"url":"https://api.github.com/repos/moby/moby/labels/area/runtime","name":"area/runtime","color":"fbca04","default":false},{"id":190258329,"url":"https://api.github.com/repos/moby/moby/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-03-21T21:08:19Z","updated_at":"2018-03-22T19:35:04Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Description**\r\n\r\nOccasionally the Docker daemon stops responding to interactions with a containers that have HEALTHCHECKs. The problem presents itself in [several older versions of Docker and the latest packaged versions on Ubuntu and v17.12.0-ce on Amazon Linux](https://github.com/jahkeup/health-stats-repro#tested-against).\r\n\r\nBy our observations, this looks to be a race condition that is met with a deadlock that prevents further calls against affected containers.\r\n\r\nThis observed issue may be related to https://github.com/moby/moby/issues/35933 . I'm working on bisecting the releases (using https://github.com/docker/docker-ce) using this repro to narrow down the problem further in any case.\r\n\r\n**Steps to reproduce the issue:**\r\n\r\nA repro case has been [built and run against several version of docker](https://github.com/jahkeup/health-stats-repro) with positive results after a few rounds of execution (I recommend 10-20 rounds to tickle the bug). There likely isn't anything specific about the 2 containers, but it has been positively triggering the bug for this test.\r\n\r\n1. Build container image with HEALTHCHECK defined (`echo hello` every `1s`)\r\n2. Start 2 containers using image\r\n3. Wait some time (`10s` in our test)\r\n4. Stop containers\r\n5. Inspect containers\r\n\r\n**Describe the results you received:**\r\n\r\nStarted containers appear to continue running and to be healthy despite being non-responsive. \r\n\r\n```bash\r\nubuntu@ip-172-31-37-156:~$ docker ps\r\nCONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS                    PORTS               NAMES\r\n0cf518c205f7        docker-poke:healthchecks   \"sh -c 'sleep 30m'\"   17 minutes ago      Up 17 minutes (healthy)                       sad_hugle\r\nubuntu@ip-172-31-37-156:~$ docker inspect 0cf518c205f7\r\n^C\r\n```\r\n\r\nAdditionally, the output of `docker ps`  will continue reporting that the container is still up and running even though the process will exit after 30m (started with `sleep 30m`).\r\n\r\n```bash\r\n0cf518c205f7        docker-poke:healthchecks   \"sh -c 'sleep 30m'\"   37 minutes ago      Up 37 minutes (healthy)                               sad_hugle\r\n```\r\n\r\n\r\n**Describe the results you expected:**\r\n\r\nI expected that I would be able to inspect this container.\r\n\r\n```bash\r\ndocker inspect 0cf518c205f7\r\n{\r\n   ...\r\n}\r\n```\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\nThis issue is readily made apparent with a few concurrent runs, but otherwise lies dormant even with many serial runs.\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:       17.12.1-ce\r\n API version:   1.35\r\n Go version:    go1.9.4\r\n Git commit:    7390fc6\r\n Built: Tue Feb 27 22:17:40 2018\r\n OS/Arch:       linux/amd64\r\n\r\nServer:\r\n Engine:\r\n  Version:      17.12.1-ce\r\n  API version:  1.35 (minimum version 1.12)\r\n  Go version:   go1.9.4\r\n  Git commit:   7390fc6\r\n  Built:        Tue Feb 27 22:16:13 2018\r\n  OS/Arch:      linux/amd64\r\n  Experimental: false\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 1\r\n Running: 1\r\n Paused: 0\r\n Stopped: 0\r\nImages: 5\r\nServer Version: 17.12.1-ce\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 4\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 9b55aab90508bd389d7654c4baf173a981477d55\r\nrunc version: 9f9c96235cc97674e935002fc3d78361b696a69e\r\ninit version: 949e6fa\r\nSecurity Options:\r\n apparmor\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.4.0-1052-aws\r\nOperating System: Ubuntu 16.04.4 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 3.625GiB\r\nName: ip-172-31-37-156\r\nID: L4W4:V4WA:OHSS:QTGL:DRJG:32GX:7DKK:FFLO:WKR2:IJYV:NKDG:GWRA\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\nWARNING: No swap limit support\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nUbuntu 16.04 on AWS EC2 using the [repro runner](https://github.com/jahkeup/health-stats-repro#ubuntu)\r\n\r\nThanks @samuelkarp!\r\n\r\n| Package                   | Result |\r\n|---------------------------|--------|\r\n| `17.09.1~ce-0~ubuntu`     | pass   |\r\n| `17.10.0~ce-0~ubuntu`     | pass   |\r\n| `17.11.0~ce-0~ubuntu`     | pass   |\r\n| `17.12.0~ce~rc1-0~ubuntu` | fail   |\r\n| `17.12.0~ce-0~ubuntu`     | fail   |\r\n| `17.12.1~ce-0~ubuntu`     | fail   |\r\n| `18.01.0~ce-0~ubuntu`     | fail   |\r\n| `18.02.0~ce-0~ubuntu`     | fail   |\r\n| `18.03.0~ce~rc4-0~ubuntu` | fail   |\r\n\r\nAmazon Linux on AWS EC2 using the [repro runner](https://github.com/jahkeup/health-stats-repro#amazon-linux)\r\n\r\nThanks @jhaynes!\r\n\r\n| Package      | Result |\r\n|--------------|--------|\r\n| `17.12.0-ce` | fail   |\r\n| `17.09.1-ce` | pass   |"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/375431200","html_url":"https://github.com/moby/moby/issues/36661#issuecomment-375431200","issue_url":"https://api.github.com/repos/moby/moby/issues/36661","id":375431200,"user":{"login":"jahkeup","id":1139752,"avatar_url":"https://avatars2.githubusercontent.com/u/1139752?v=4","gravatar_id":"","url":"https://api.github.com/users/jahkeup","html_url":"https://github.com/jahkeup","followers_url":"https://api.github.com/users/jahkeup/followers","following_url":"https://api.github.com/users/jahkeup/following{/other_user}","gists_url":"https://api.github.com/users/jahkeup/gists{/gist_id}","starred_url":"https://api.github.com/users/jahkeup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jahkeup/subscriptions","organizations_url":"https://api.github.com/users/jahkeup/orgs","repos_url":"https://api.github.com/users/jahkeup/repos","events_url":"https://api.github.com/users/jahkeup/events{/privacy}","received_events_url":"https://api.github.com/users/jahkeup/received_events","type":"User","site_admin":false},"created_at":"2018-03-22T19:35:04Z","updated_at":"2018-03-22T19:35:04Z","author_association":"CONTRIBUTOR","body":"The attached output from the above shim has a deleted FIFO for its stdout:\r\n\r\n```\r\nroot@ip-172-31-47-105:/proc/1286/fd# lsof -p 1286\r\nCOMMAND    PID USER   FD      TYPE             DEVICE SIZE/OFF    NODE NAME\r\ndocker-co 1286 root  cwd       DIR               0,19      140    1849 /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/18d45667967f6c29dd86ae36b85130416f1afefa6537bbdbc66a5f5a029c58ad\r\ndocker-co 1286 root  rtd       DIR              259,1     4096       2 /\r\ndocker-co 1286 root  txt       REG              259,1  4324256   51295 /usr/bin/docker-containerd-shim\r\ndocker-co 1286 root    0r      CHR                1,3      0t0       6 /dev/null\r\ndocker-co 1286 root    1u     unix 0xffff8803a851ec00      0t0 1463760 type=STREAM\r\ndocker-co 1286 root    2u     unix 0xffff8803a851ec00      0t0 1463760 type=STREAM\r\ndocker-co 1286 root    4u  a_inode               0,11        0    8025 [eventpoll]\r\ndocker-co 1286 root    5u  a_inode               0,11        0    8025 [eventpoll]\r\ndocker-co 1286 root    6u     unix 0xffff880360544800      0t0 2019410 @/containerd-shim/moby/18d45667967f6c29dd86ae36b85130416f1afefa6537bbdbc66a5f5a029c58ad/shim.sock type=STREAM\r\ndocker-co 1286 root    7u     unix 0xffff880387a35c00      0t0 2016007 @/containerd-shim/moby/18d45667967f6c29dd86ae36b85130416f1afefa6537bbdbc66a5f5a029c58ad/shim.sock type=STREAM\r\ndocker-co 1286 root    8r     FIFO               0,10      0t0 2016009 pipe\r\ndocker-co 1286 root    9r     FIFO               0,10      0t0 2088440 pipe\r\ndocker-co 1286 root   10r     FIFO               0,10      0t0 2016010 pipe\r\ndocker-co 1286 root   12r     FIFO               0,10      0t0 2016011 pipe\r\ndocker-co 1286 root   20w     FIFO               0,10      0t0 2088440 pipe\r\ndocker-co 1286 root   21r     FIFO               0,10      0t0 2088441 pipe\r\ndocker-co 1286 root   22u     FIFO               0,19      0t0     640 /run/docker/containerd/18d45667967f6c29dd86ae36b85130416f1afefa6537bbdbc66a5f5a029c58ad/be86934cf866422f105bb87b7b4938d5445b23d1f44aa8a6dee909793393a865-stdout (deleted)\r\ndocker-co 1286 root   23r     FIFO               0,10      0t0 2088442 pipe\r\n```\r\n\r\nThere's no difference between the above where the container is attempted to be stopped and removed, and the below where the container is started and then inspected (no stop or remove attempts):\r\n\r\n```\r\nCOMMAND    PID USER   FD      TYPE             DEVICE SIZE/OFF   NODE NAME\r\ndocker-co 1331 root  cwd       DIR               0,19      140   1515 /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/42930b0c577475cb4a77e8be0d54f0e8a8eb026c4ccff14d69b4e68b80e89619\r\ndocker-co 1331 root  rtd       DIR              259,1     4096      2 /\r\ndocker-co 1331 root  txt       REG              259,1  4324256  51295 /usr/bin/docker-containerd-shim\r\ndocker-co 1331 root    0r      CHR                1,3      0t0      6 /dev/null\r\ndocker-co 1331 root    1w      CHR                1,3      0t0      6 /dev/null\r\ndocker-co 1331 root    2w      CHR                1,3      0t0      6 /dev/null\r\ndocker-co 1331 root    3r     FIFO               0,10      0t0 390245 pipe\r\ndocker-co 1331 root    4u  a_inode               0,11        0   8025 [eventpoll]\r\ndocker-co 1331 root    5u  a_inode               0,11        0   8025 [eventpoll]\r\ndocker-co 1331 root    6u     unix 0xffff880391a29c00      0t0  80731 @/containerd-shim/moby/42930b0c577475cb4a77e8be0d54f0e8a8eb026c4ccff14d69b4e68b80e89619/shim.sock type=STREAM\r\ndocker-co 1331 root    7u     unix 0xffff880394c01c00      0t0  80735 @/containerd-shim/moby/42930b0c577475cb4a77e8be0d54f0e8a8eb026c4ccff14d69b4e68b80e89619/shim.sock type=STREAM\r\ndocker-co 1331 root    8r     FIFO               0,10      0t0  80737 pipe\r\ndocker-co 1331 root    9w     FIFO               0,10      0t0 390245 pipe\r\ndocker-co 1331 root   10r     FIFO               0,10      0t0  80738 pipe\r\ndocker-co 1331 root   12r     FIFO               0,10      0t0  80739 pipe\r\ndocker-co 1331 root   20r     FIFO               0,10      0t0 390246 pipe\r\ndocker-co 1331 root   21u     FIFO               0,19      0t0   3262 /run/docker/containerd/42930b0c577475cb4a77e8be0d54f0e8a8eb026c4ccff14d69b4e68b80e89619/9b200a3c4f15d44fd029699fcca2d5ccece43f2c7dc6d5edeb198e4eb4ec251b-stdout (deleted)\r\ndocker-co 1331 root   22r     FIFO               0,10      0t0 390247 pipe\r\n```"}}
{"actor":{"display_login":"jahkeup"}}