{"action":"closed","issue":{"url":"https://api.github.com/repos/moby/moby/issues/36002","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/36002/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/36002/comments","events_url":"https://api.github.com/repos/moby/moby/issues/36002/events","html_url":"https://github.com/moby/moby/issues/36002","id":288181868,"number":36002,"title":"Killing docker-containerd breaks interaction with containers","user":{"login":"thaJeztah","id":1804568,"avatar_url":"https://avatars2.githubusercontent.com/u/1804568?v=4","gravatar_id":"","url":"https://api.github.com/users/thaJeztah","html_url":"https://github.com/thaJeztah","followers_url":"https://api.github.com/users/thaJeztah/followers","following_url":"https://api.github.com/users/thaJeztah/following{/other_user}","gists_url":"https://api.github.com/users/thaJeztah/gists{/gist_id}","starred_url":"https://api.github.com/users/thaJeztah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thaJeztah/subscriptions","organizations_url":"https://api.github.com/users/thaJeztah/orgs","repos_url":"https://api.github.com/users/thaJeztah/repos","events_url":"https://api.github.com/users/thaJeztah/events{/privacy}","received_events_url":"https://api.github.com/users/thaJeztah/received_events","type":"User","site_admin":false},"labels":[{"id":73641990,"url":"https://api.github.com/repos/moby/moby/labels/area/runtime","name":"area/runtime","color":"fbca04","default":false},{"id":190258329,"url":"https://api.github.com/repos/moby/moby/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false},{"id":695263242,"url":"https://api.github.com/repos/moby/moby/labels/version/18.01","name":"version/18.01","color":"ededed","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-01-12T16:54:03Z","updated_at":"2018-02-08T14:19:30Z","closed_at":"2018-02-08T14:19:30Z","author_association":"MEMBER","body":"When killing `docker-containerd`, interacting with containers (`docker exec`, `docker stop`, `docker kill`) fails:\r\n\r\n```bash\r\ndocker kill testing\r\nError response from daemon: Cannot kill container: testing: Cannot kill container 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359: connection error: desc = \"transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused\": unknown\r\n\r\ndocker rm -f lucid_yalow\r\nError response from daemon: Could not kill running container 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359, cannot remove - Cannot kill container 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359: connection error: desc = \"transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused\": unknown\r\n```\r\n\r\nBut killing `dockerd` (either by `killall -9 dockerd` or a `SIGHUP`; `killall -HUP dockerd`) restores functionality.\r\n\r\nThis problem could explain some reports about \"unkillable\" containers, where everything appears to be running, but interaction is not possible (possibly after `containerd` was OOM killed, but could have different causes).\r\n\r\n\r\n### Steps to reproduce / information\r\n\r\nHave docker running, start a container, and check output of `ps auxf`: `docker-containerd` and `docker-containerd-shim` are child-processes of `dockerd`:\r\n\r\n```\r\nroot     11468  1.1  3.4 468232 71036 ?        Ssl  11:56   0:01 /usr/bin/dockerd -H fd://\r\nroot     11473  0.4  1.3 236512 27856 ?        Ssl  11:56   0:00  \\_ docker-containerd --config /var/run/docker/containerd/containerd.toml\r\nroot     11918  0.0  0.1   7516  3788 ?        Sl   11:57   0:00      \\_ docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc\r\nroot     11933  0.1  0.0   1236     4 pts/0    Ss+  11:57   0:00          \\_ sh\r\n```\r\n\r\n\r\nNow, kill `docker-containerd` (`killall -9 docker-containerd`).\r\n\r\n`docker-containerd` is restarted (by `dockerd`); observe that `docker-containerd-shim` and the container process(es) are reparented (I haven't checked what the new parent process is, and if this is relevant). The `docker-containerd-shim` processes are no longer child-process of `docker-containerd`;\r\n\r\n```\r\nroot     11468  160  3.6 470984 74664 ?        Ssl  11:56  19:55 /usr/bin/dockerd -H fd://\r\nroot     11979  0.1  1.2 300992 25980 ?        Ssl  11:58   0:01  \\_ docker-containerd --config /var/run/docker/containerd/containerd.toml\r\nroot     11918  0.0  0.2   7516  4688 ?        Sl   11:57   0:00 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc\r\nroot     11933  0.0  0.0   1236     4 pts/0    Ss+  11:57   0:00  \\_ sh\r\n```\r\n\r\nAt this point, interacting with containers is now broken..\r\n\r\n\r\nContainers still show up as running:\r\n\r\n```bash\r\ndocker ps\r\n\r\nCONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES\r\n9bfdba3fc8ee        busybox             \"sh\"                About a minute ago   Up About a minute                       testing\r\n```\r\n\r\nInspecting the container still works, and shows the `pid` of the container;\r\n\r\n```bash\r\ndocker inspect --format '{{json .State}}' testing | jq .\r\n\r\n{\r\n  \"Status\": \"running\",\r\n  \"Running\": true,\r\n  \"Paused\": false,\r\n  \"Restarting\": false,\r\n  \"OOMKilled\": false,\r\n  \"Dead\": false,\r\n  \"Pid\": 11933,\r\n  \"ExitCode\": 0,\r\n  \"Error\": \"\",\r\n  \"StartedAt\": \"2018-01-12T11:57:47.687627373Z\",\r\n  \"FinishedAt\": \"0001-01-01T00:00:00Z\"\r\n}\r\n```\r\n\r\n\r\nBut any interaction with the containers is broken;\r\n\r\n```bash\r\ndocker kill testing\r\nError response from daemon: Cannot kill container: testing: Cannot kill container 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359: connection error: desc = \"transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused\": unknown\r\n\r\ndocker rm -f lucid_yalow\r\nError response from daemon: Could not kill running container 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359, cannot remove - Cannot kill container 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359: connection error: desc = \"transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused\": unknown\r\n```\r\n\r\n\r\nWhen directly connecting to containerd, containers still show:\r\n\r\n```bash\r\ndocker-containerd-ctr --namespace=moby --address /var/run/docker/containerd/docker-containerd.sock containers ls\r\n\r\nCONTAINER                                                           IMAGE    RUNTIME                           \r\n9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359    -        io.containerd.runtime.v1.linux    \r\n```\r\n\r\nAnd can be inspected;\r\n\r\n```bash\r\ndocker-containerd-ctr --namespace=moby --address /var/run/docker/containerd/docker-containerd.sock containers info 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359\r\n\r\n......\r\n```\r\n\r\nShims are still up:\r\n\r\n```\r\nnetstat -x | grep shim\r\nunix  2      [ ]         STREAM     CONNECTED     64641    @/containerd-shim/moby/9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359/shim.sock\r\nunix  3      [ ]         STREAM     CONNECTED     64019    @/containerd-shim/moby/9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359/shim.sock\r\n```\r\n\r\n\r\n```bash\r\ndocker-runc --root /var/run/docker/runtime-runc/moby/ state 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359\r\n{\r\n  \"ociVersion\": \"1.0.0\",\r\n  \"id\": \"9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359\",\r\n  \"pid\": 11933,\r\n  \"status\": \"running\",\r\n  \"bundle\": \"/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359\",\r\n  \"rootfs\": \"/var/lib/docker/overlay2/9c0e355304db9fb85f7c1281b11008eea23bd4dbb142f11f551066c9fdb2e70e/merged\",\r\n  \"created\": \"2018-01-12T11:57:47.631870877Z\",\r\n  \"owner\": \"\"\r\n}\r\n```\r\n\r\n\r\nAnd the container is still functional, when using `docker-runc`;\r\n\r\n```bash\r\ndocker-runc --root /var/run/docker/runtime-runc/moby/ exec 9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359 ls -la\r\n\r\ntotal 44\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 11:57 .\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 11:57 ..\r\n-rwxr-xr-x    1 root     root             0 Jan 12 11:57 .dockerenv\r\ndrwxr-xr-x    2 root     root         12288 Jan  8 21:14 bin\r\ndrwxr-xr-x    5 root     root           360 Jan 12 11:57 dev\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 11:57 etc\r\ndrwxr-xr-x    2 nobody   nogroup       4096 Jan  8 21:14 home\r\ndr-xr-xr-x  125 root     root             0 Jan 12 11:57 proc\r\ndrwxr-xr-x    2 root     root          4096 Jan  8 21:14 root\r\ndr-xr-xr-x   13 root     root             0 Jan 12 11:57 sys\r\ndrwxrwxrwt    2 root     root          4096 Jan  8 21:14 tmp\r\ndrwxr-xr-x    3 root     root          4096 Jan  8 21:14 usr\r\ndrwxr-xr-x    4 root     root          4096 Jan  8 21:14 var\r\n```\r\n\r\n### restore functionality\r\n\r\nKill `dockerd` (`killall -9 dockerd`) or `SIGHUP` (`killall -HUP dockerd`).\r\n\r\nObserve that shims are not re-parented (which is probably expected);\r\n\r\n```\r\nroot     11918  0.0  0.2   7516  4688 ?        Sl   11:57   0:00 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9bfdba3fc8eee79d6ca5773f7caff5dc5a8379037e98b6ded5c8b68df5750359 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-contai\r\nroot     11933  0.0  0.0   1236     4 pts/0    Ss+  11:57   0:00  \\_ sh\r\nroot     12287  1.1  2.8 446232 57824 ?        Ssl  12:55   0:00 /usr/bin/dockerd -H fd://\r\nroot     12293  0.7  1.1 300928 22616 ?        Ssl  12:55   0:00  \\_ docker-containerd --config /var/run/docker/containerd/containerd.toml\r\n```\r\n\r\n\r\nBut now it's possible again to interact with them:\r\n\r\n```\r\ndocker exec testing ls -la\r\n\r\ntotal 44\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 11:57 .\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 11:57 ..\r\n-rwxr-xr-x    1 root     root             0 Jan 12 11:57 .dockerenv\r\ndrwxr-xr-x    2 root     root         12288 Jan  8 21:14 bin\r\ndrwxr-xr-x    5 root     root           360 Jan 12 11:57 dev\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 11:57 etc\r\ndrwxr-xr-x    2 nobody   nogroup       4096 Jan  8 21:14 home\r\ndr-xr-xr-x  126 root     root             0 Jan 12 11:57 proc\r\ndrwxr-xr-x    1 root     root          4096 Jan 12 12:58 root\r\ndr-xr-xr-x   13 root     root             0 Jan 12 11:57 sys\r\ndrwxrwxrwt    2 root     root          4096 Jan  8 21:14 tmp\r\ndrwxr-xr-x    3 root     root          4096 Jan  8 21:14 usr\r\ndrwxr-xr-x    4 root     root          4096 Jan  8 21:14 var\r\n```\r\n\r\n## Version of docker and containerd\r\n\r\n\r\nTested on Ubuntu 16.04 on DigitalOcean;\r\n\r\n```\r\ndocker-containerd --version\r\ncontainerd github.com/containerd/containerd v1.0.0 89623f28b87a6004d4b785663257362d1658a729\r\n```\r\n\r\n```\r\nClient:\r\n Version:\t18.01.0-ce\r\n API version:\t1.35\r\n Go version:\tgo1.9.2\r\n Git commit:\t03596f5\r\n Built:\tWed Jan 10 20:11:05 2018\r\n OS/Arch:\tlinux/amd64\r\n Experimental:\tfalse\r\n Orchestrator:\tswarm\r\n\r\nServer:\r\n Engine:\r\n  Version:\t18.01.0-ce\r\n  API version:\t1.35 (minimum version 1.12)\r\n  Go version:\tgo1.9.2\r\n  Git commit:\t03596f5\r\n  Built:\tWed Jan 10 20:09:37 2018\r\n  OS/Arch:\tlinux/amd64\r\n  Experimental:\tfalse\r\n```\r\n\r\n\r\n```\r\nContainers: 1\r\n Running: 1\r\n Paused: 0\r\n Stopped: 0\r\nImages: 2\r\nServer Version: 18.01.0-ce\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 89623f28b87a6004d4b785663257362d1658a729\r\nrunc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f\r\ninit version: 949e6fa\r\nSecurity Options:\r\n apparmor\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.4.0-108-generic\r\nOperating System: Ubuntu 16.04.3 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 1.953GiB\r\nName: ubuntu-2gb-ams3-01\r\nID: KIY5:X5P2:5FI5:GEPC:Q2OO:XF4P:KFB2:S22T:A76T:DVFV:UIFB:ZATY\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\nWARNING: No swap limit support\r\n```\r\n\r\n"}}
{"actor":{"display_login":"yongtang"}}