{"action":"closed","issue":{"url":"https://api.github.com/repos/docker/docker/issues/21893","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/21893/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/21893/comments","events_url":"https://api.github.com/repos/docker/docker/issues/21893/events","html_url":"https://github.com/docker/docker/issues/21893","id":147007769,"number":21893,"title":"Docker container is unremovable (docker-in-docker, aufs)","user":{"login":"umitanuki","id":33334,"avatar_url":"https://avatars.githubusercontent.com/u/33334?v=3","gravatar_id":"","url":"https://api.github.com/users/umitanuki","html_url":"https://github.com/umitanuki","followers_url":"https://api.github.com/users/umitanuki/followers","following_url":"https://api.github.com/users/umitanuki/following{/other_user}","gists_url":"https://api.github.com/users/umitanuki/gists{/gist_id}","starred_url":"https://api.github.com/users/umitanuki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/umitanuki/subscriptions","organizations_url":"https://api.github.com/users/umitanuki/orgs","repos_url":"https://api.github.com/users/umitanuki/repos","events_url":"https://api.github.com/users/umitanuki/events{/privacy}","received_events_url":"https://api.github.com/users/umitanuki/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/area/runtime","name":"area/runtime","color":"e11d21"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2016-04-08T18:52:08Z","updated_at":"2016-04-16T09:27:11Z","closed_at":"2016-04-16T09:27:11Z","body":"I'm running docker-in-docker (not sure if bare docker hits the issue too) on aufs storage driver, and frequently (more than once a day) one of the containers become unremovable.  More info:\r\n\r\ndocker info\r\n```\r\ndocker info\r\nContainers: 57\r\n Running: 0\r\n Paused: 0\r\n Stopped: 57\r\nImages: 8\r\nServer Version: 1.10.3\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 194\r\n Dirperm1 Supported: true\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nPlugins:\r\n Volume: local\r\n Network: bridge null host\r\nKernel Version: 3.19.0-56-generic\r\nOperating System: Alpine Linux v3.3\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 24\r\nTotal Memory: 62.82 GiB\r\nName: 3470421a6dc6\r\nID: YMAO:PRWC:6LQS:VPSR:KXWR:ZD45:ROG7:L322:5K2N:MBXH:SNHZ:FWTW\r\nWARNING: No swap limit support\r\nWARNING: bridge-nf-call-iptables is disabled\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\n```\r\n\r\nParent docker\r\n```\r\n$ DOCKER_HOST= docker info\r\nContainers: 18\r\n Running: 17\r\n Paused: 0\r\n Stopped: 1\r\nImages: 8\r\nServer Version: 1.10.0\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 71\r\n Dirperm1 Supported: true\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nPlugins:\r\n Volume: local\r\n Network: bridge null host\r\nKernel Version: 3.19.0-56-generic\r\nOperating System: Ubuntu 14.04.4 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 24\r\nTotal Memory: 62.82 GiB\r\nName: slk80-5\r\nID: UU7P:A2L6:W3YK:MBJR:DT5L:7E26:PJYV:4SUB:QKMX:RNM3:V7NA:FS5I\r\nWARNING: No swap limit support\r\n```\r\n\r\ndocker version\r\n```\r\n$ docker version\r\nClient:\r\n Version:      1.10.0\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   590d5108\r\n Built:        Thu Feb  4 18:36:33 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.10.3\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   20f81dd\r\n Built:        Thu Mar 10 21:49:11 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\nParent docker version\r\n```\r\n$ DOCKER_HOST= docker version\r\nClient:\r\n Version:      1.10.0\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   590d5108\r\n Built:        Thu Feb  4 18:36:33 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.10.0\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   590d5108\r\n Built:        Thu Feb  4 18:36:33 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n** Symptom\r\nOne of the containers become unremovable.\r\n\r\n```\r\n$ docker ps\r\nCONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES\r\n9a1d2d5b2665        www                 \"supervisord -c /proj\"   14 hours ago        Up 14 hours                             livetester\r\n$ docker rm livetester\r\nlivetester\r\n$ docker rm livetester\r\nlivetester\r\n$ docker ps\r\nCONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES\r\n9a1d2d5b2665        www                 \"supervisord -c /proj\"   14 hours ago        Up 14 hours                             livetester\r\n```\r\n\r\nNote that even though docker says it's removed, container still exists in `ps`.\r\n\r\nThe process doesn't exist.  It's just container is still in `ps`.\r\n\r\n```\r\n$ docker inspect livetester\r\n[\r\n    {\r\n        \"Id\": \"9a1d2d5b266547383ed266b7f897dd7fdd3f7feae943d61d943c4c35fb9d231e\",\r\n        \"Created\": \"2016-04-08T04:39:21.426447657Z\",\r\n        \"Path\": \"supervisord\",\r\n        \"Args\": [\r\n            \"-c\",\r\n            \"/project/dockerfiles/livetester/supervisord.conf\"\r\n        ],\r\n        \"State\": {\r\n            \"Status\": \"running\",\r\n            \"Running\": true,\r\n            \"Paused\": false,\r\n            \"Restarting\": false,\r\n            \"OOMKilled\": false,\r\n            \"Dead\": false,\r\n            \"Pid\": 15431,\r\n            \"ExitCode\": 0,\r\n            \"Error\": \"\",\r\n            \"StartedAt\": \"2016-04-08T04:39:22.777812586Z\",\r\n            \"FinishedAt\": \"0001-01-01T00:00:00Z\"\r\n        },\r\n        \"Image\": \"sha256:a13efacf382c9d1ecb4b4c3815e2dd97e03628fab8e8ce6926caa18596b2ab7e\",\r\n        \"ResolvConfPath\": \"/var/lib/docker/containers/7638ab782e64d341f6c9d7a06b53f43b098823e68c4a3f63c3e8d9b628d1b02b/resolv.conf\",\r\n        \"HostnamePath\": \"/var/lib/docker/containers/7638ab782e64d341f6c9d7a06b53f43b098823e68c4a3f63c3e8d9b628d1b02b/hostname\",\r\n        \"HostsPath\": \"/var/lib/docker/containers/7638ab782e64d341f6c9d7a06b53f43b098823e68c4a3f63c3e8d9b628d1b02b/hosts\",\r\n        \"LogPath\": \"/var/lib/docker/containers/9a1d2d5b266547383ed266b7f897dd7fdd3f7feae943d61d943c4c35fb9d231e/9a1d2d5b266547383ed266b7f897dd7fdd3f7feae943d61d943c4c35fb9d231e-json.log\",\r\n        \"Name\": \"/livetester\",\r\n        \"RestartCount\": 0,\r\n        \"Driver\": \"aufs\",\r\n        \"MountLabel\": \"\",\r\n        \"ProcessLabel\": \"\",\r\n        \"AppArmorProfile\": \"\",\r\n        \"ExecIDs\": null,\r\n        \"HostConfig\": {\r\n            \"Binds\": [\r\n                \"/home/stg-app-0/deploy/dockerfiles/livetester/../..:/project\",\r\n                \"/etc/localtime:/etc/localtime:ro\",\r\n                \"/var/multi/stg-app-0/data:/project/data\"\r\n            ],\r\n            \"ContainerIDFile\": \"\",\r\n            \"LogConfig\": {\r\n                \"Type\": \"json-file\",\r\n                \"Config\": {}\r\n            },\r\n            \"NetworkMode\": \"container:net\",\r\n            \"PortBindings\": {},\r\n            \"RestartPolicy\": {\r\n                \"Name\": \"no\",\r\n                \"MaximumRetryCount\": 0\r\n            },\r\n            \"VolumeDriver\": \"\",\r\n            \"VolumesFrom\": null,\r\n            \"CapAdd\": null,\r\n            \"CapDrop\": null,\r\n            \"Dns\": [],\r\n            \"DnsOptions\": [],\r\n            \"DnsSearch\": [],\r\n            \"ExtraHosts\": null,\r\n            \"GroupAdd\": null,\r\n            \"IpcMode\": \"\",\r\n            \"Links\": null,\r\n            \"OomScoreAdj\": 0,\r\n            \"PidMode\": \"\",\r\n            \"Privileged\": false,\r\n            \"PublishAllPorts\": false,\r\n            \"ReadonlyRootfs\": false,\r\n            \"SecurityOpt\": null,\r\n            \"UTSMode\": \"\",\r\n            \"ShmSize\": 67108864,\r\n            \"ConsoleSize\": [\r\n                0,\r\n                0\r\n            ],\r\n            \"Isolation\": \"\",\r\n            \"CpuShares\": 0,\r\n            \"CgroupParent\": \"\",\r\n            \"BlkioWeight\": 0,\r\n            \"BlkioWeightDevice\": null,\r\n            \"BlkioDeviceReadBps\": null,\r\n            \"BlkioDeviceWriteBps\": null,\r\n            \"BlkioDeviceReadIOps\": null,\r\n            \"BlkioDeviceWriteIOps\": null,\r\n            \"CpuPeriod\": 0,\r\n            \"CpuQuota\": 0,\r\n            \"CpusetCpus\": \"\",\r\n            \"CpusetMems\": \"\",\r\n            \"Devices\": [],\r\n            \"KernelMemory\": 0,\r\n            \"Memory\": 0,\r\n            \"MemoryReservation\": 0,\r\n            \"MemorySwap\": 0,\r\n            \"MemorySwappiness\": -1,\r\n            \"OomKillDisable\": false,\r\n            \"PidsLimit\": 0,\r\n            \"Ulimits\": null\r\n        },\r\n        \"GraphDriver\": {\r\n            \"Name\": \"aufs\",\r\n            \"Data\": null\r\n        },\r\n        \"Mounts\": [\r\n            {\r\n                \"Source\": \"/home/stg-app-0/deploy\",\r\n                \"Destination\": \"/project\",\r\n                \"Mode\": \"\",\r\n                \"RW\": true,\r\n                \"Propagation\": \"rprivate\"\r\n            },\r\n            {\r\n                \"Source\": \"/etc/localtime\",\r\n                \"Destination\": \"/etc/localtime\",\r\n                \"Mode\": \"ro\",\r\n                \"RW\": false,\r\n                \"Propagation\": \"rprivate\"\r\n            },\r\n            {\r\n                \"Source\": \"/var/multi/stg-app-0/data\",\r\n                \"Destination\": \"/project/data\",\r\n                \"Mode\": \"\",\r\n                \"RW\": true,\r\n                \"Propagation\": \"rprivate\"\r\n            }\r\n        ],\r\n        \"Config\": {\r\n            \"Hostname\": \"7638ab782e64\",\r\n            \"Domainname\": \"\",\r\n            \"User\": \"\",\r\n            \"AttachStdin\": false,\r\n            \"AttachStdout\": false,\r\n            \"AttachStderr\": false,\r\n            \"ExposedPorts\": {\r\n                \"5555/tcp\": {},\r\n                \"5556/tcp\": {},\r\n                \"5557/tcp\": {}\r\n            },\r\n            \"Tty\": false,\r\n            \"OpenStdin\": false,\r\n            \"StdinOnce\": false,\r\n            \"Env\": [\r\n                \"HISTFILE=/project/data/.bash_history\",\r\n                \"HOSTNAME=slk80-5-stg-app-0-livetester\",\r\n                \"GAURUN_PROD_PORT=4500\",\r\n                \"WORKER_FLOWER_DEV_PORT=5556\",\r\n                \"GAURUN_HOST=localhost\",\r\n                \"WORKER_FLOWER_PROD_PORT=5557\",\r\n                \"NUM_TRAINING_WORKERS_CORE_3=1\",\r\n                \"NUM_TRAINING_WORKERS_CORE_2=1\",\r\n                \"ALGOMANAGER_ETCD_TTL=60\",\r\n                \"NUM_TRAINING_WORKERS_CORE_0=1\",\r\n                \"CAHRT_GENERATION_PORT=4100\",\r\n                \"POSTGRESQL_HOST=localhost\",\r\n                \"GPU_ID=1\",\r\n                \"FLASK_HTTP_PORT=5000\",\r\n                \"APP_CONFIG=StgConfig\",\r\n                \"WORKER_FLOWER_TEST_PORT=5555\",\r\n                \"DATAPROVIDER_MAX_PROCS=32\",\r\n                \"TRADESERVER_HOST=localhost\",\r\n                \"WS_PORT=4000\",\r\n                \"IS_PROD=False\",\r\n                \"ETCD_HOST=127.0.0.1\",\r\n                \"REDIS_PORT=6379\",\r\n                \"DATASOURCE_ENDPOINT=http://globalcurrencies.xignite.com/xGlobalCurrencies.json\",\r\n                \"DATACLIENT_RETRIES=100\",\r\n                \"POSTGRESQL_PORT=5432\",\r\n                \"NUM_TRAINING_WORKERS_CORE_1=1\",\r\n                \"EXTRA_IF_HOST=eth4\",\r\n                \"CAHRT_GENERATION_HOST=localhost\",\r\n                \"NUM_SCORING_WORKERS_CORE_3=1\",\r\n                \"NUM_SCORING_WORKERS_CORE_2=1\",\r\n                \"NUM_SCORING_WORKERS_CORE_1=1\",\r\n                \"NUM_SCORING_WORKERS_CORE_0=1\",\r\n                \"MEMCACHED_HOST=localhost\",\r\n                \"GAURUN_DEV_SANDBOX_PORT=4800\",\r\n                \"LOG_LEVEL=20\",\r\n                \"DATACLIENT_TIMEOUT=15000\",\r\n                \"EXTRA_IF_GW=10.122.62.1\",\r\n                \"MEMCACHED_PORT=11211\",\r\n                \"ETCD_INITIAL_CLUSTER=\",\r\n                \"ALGOMANAGER_ADMIN_PORT=5999\",\r\n                \"REDIS_MEMCACHE_DB=15\",\r\n                \"BASE_URL=https://staging.capitalico.com\",\r\n                \"TRADESERVER_PORT=6000\",\r\n                \"REDIS_HOST=localhost\",\r\n                \"DATASOURCE_TOKEN=A52B79B807E24AFD9A009A85D6AF473B\",\r\n                \"ALPACANITE_PORT=2222\",\r\n                \"GAURUN_DEV_PROD_PORT=4700\",\r\n                \"GAURUN_SANDBOX_PORT=4600\",\r\n                \"EXTRA_IF_IPMASK=10.122.62.20/26\",\r\n                \"DATAPROVIDER_HOST=10.122.62.30\",\r\n                \"ETCD_PORT=2379\",\r\n                \"FAKEMODE=FALSE\",\r\n                \"PATH=/project/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/cuda/bin:/opt/gaurun:/opt/chart\",\r\n                \"LD_LIBRARY_PATH=/usr/local/lib:/usr/local/cuda/lib64:\",\r\n                \"PYTHONPATH=/project/python/app:/project/python/lib\"\r\n            ],\r\n            \"Cmd\": [\r\n                \"supervisord\",\r\n                \"-c\",\r\n                \"/project/dockerfiles/livetester/supervisord.conf\"\r\n            ],\r\n            \"Image\": \"www\",\r\n            \"Volumes\": null,\r\n            \"WorkingDir\": \"/project\",\r\n            \"Entrypoint\": null,\r\n            \"OnBuild\": null,\r\n            \"Labels\": {},\r\n            \"StopSignal\": \"SIGTERM\"\r\n        },\r\n        \"NetworkSettings\": {\r\n            \"Bridge\": \"\",\r\n            \"SandboxID\": \"\",\r\n            \"HairpinMode\": false,\r\n            \"LinkLocalIPv6Address\": \"\",\r\n            \"LinkLocalIPv6PrefixLen\": 0,\r\n            \"Ports\": null,\r\n            \"SandboxKey\": \"\",\r\n            \"SecondaryIPAddresses\": null,\r\n            \"SecondaryIPv6Addresses\": null,\r\n            \"EndpointID\": \"\",\r\n            \"Gateway\": \"\",\r\n            \"GlobalIPv6Address\": \"\",\r\n            \"GlobalIPv6PrefixLen\": 0,\r\n            \"IPAddress\": \"\",\r\n            \"IPPrefixLen\": 0,\r\n            \"IPv6Gateway\": \"\",\r\n            \"MacAddress\": \"\",\r\n            \"Networks\": null\r\n        }\r\n    }\r\n]\r\n```\r\n\r\nProcess 15431 doesn't exist.  The parent docker runs with `--pid=host`.\r\n\r\nNot sure if it relates, but every time I see this in the docker daemon log.\r\n\r\n```\r\ntime=\"2016-04-08T04:38:40.095336232Z\" level=error msg=\"Handler for DELETE /v1.22/containers/manual returned error: No such container: manual\"\r\ntime=\"2016-04-08T04:38:40.132337005Z\" level=error msg=\"Handler for POST /v1.22/containers/livetester/kill returned error: Cannot kill container livetester: Container 93fdd7adcfa4c732e820b08b817ca40300b915ff1030402c480a6332c391cb4c is not running\"\r\ntime=\"2016-04-08T04:38:41.722221664Z\" level=error msg=\"Couldn't run auplink before unmount: exec: \\\"auplink\\\": executable file not found in $PATH\"\r\ntime=\"2016-04-08T04:38:41.900656823Z\" level=error msg=\"Couldn't run auplink before unmount: exec: \\\"auplink\\\": executable file not found in $PATH\"\r\ntime=\"2016-04-08T04:38:58.684734947Z\" level=error msg=\"Couldn't run auplink before unmount: exec: \\\"auplink\\\": executable file not found in $PATH\"\r\ntime=\"2016-04-08T04:38:58.741898731Z\" level=error msg=\"Couldn't run auplink before unmount: exec: \\\"auplink\\\": executable file not found in $PATH\"\r\ntime=\"2016-04-08T04:39:00.014414531Z\" level=error msg=\"Couldn't run auplink before unmount: exec: \\\"auplink\\\": executable file not found in $PATH\"\r\ntime=\"2016-04-08T04:39:00.175652093Z\" level=error msg=\"Couldn't run auplink before unmount: exec: \\\"auplink\\\": executable file not found in $PATH\"\r\ntime=\"2016-04-08T04:39:00.244468344Z\" level=error msg=\"attach: stdout: write unix /sock/docker.sock->@: write: broken pipe\"\r\n```\r\n\r\nThe docker-in-docker container image doesn't have auplink apparently, but I'm not sure why this happens only after particular point.\r\n\r\n`docker start`, `docker stop` doesn't help.  `docker attach` hangs.\r\n\r\nI don't have clear repro.  It happens in different container images.\r\n\r\nAny idea?\r\n"}}