{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/28859","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/28859/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/28859/comments","events_url":"https://api.github.com/repos/docker/docker/issues/28859/events","html_url":"https://github.com/docker/docker/issues/28859","id":191826374,"number":28859,"title":"Container with mounts fails to start with --userns-remap=default option","user":{"login":"chepurko","id":206400,"avatar_url":"https://avatars.githubusercontent.com/u/206400?v=3","gravatar_id":"","url":"https://api.github.com/users/chepurko","html_url":"https://github.com/chepurko","followers_url":"https://api.github.com/users/chepurko/followers","following_url":"https://api.github.com/users/chepurko/following{/other_user}","gists_url":"https://api.github.com/users/chepurko/gists{/gist_id}","starred_url":"https://api.github.com/users/chepurko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chepurko/subscriptions","organizations_url":"https://api.github.com/users/chepurko/orgs","repos_url":"https://api.github.com/users/chepurko/repos","events_url":"https://api.github.com/users/chepurko/events{/privacy}","received_events_url":"https://api.github.com/users/chepurko/received_events","type":"User","site_admin":false},"labels":[{"id":73641990,"url":"https://api.github.com/repos/docker/docker/labels/area/runtime","name":"area/runtime","color":"fbca04","default":false},{"id":375714123,"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-11-26T20:57:34Z","updated_at":"2016-11-30T19:40:24Z","closed_at":null,"body":"<!--\r\nIf you are reporting a new issue, make sure that we do not have any duplicates\r\nalready open. You can ensure this by searching the issue list for this\r\nrepository. If there is a duplicate, please close your issue and add a comment\r\nto the existing issue instead.\r\n\r\nIf you suspect your issue is a bug, please edit your issue description to\r\ninclude the BUG REPORT INFORMATION shown below. If you fail to provide this\r\ninformation within 7 days, we cannot debug your issue and will close it. We\r\nwill, however, reopen it if you later provide the information.\r\n\r\nFor more information about reporting issues, see\r\nhttps://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues\r\n\r\n---------------------------------------------------\r\nGENERAL SUPPORT INFORMATION\r\n---------------------------------------------------\r\n\r\nThe GitHub issue tracker is for bug reports and feature requests.\r\nGeneral support can be found at the following locations:\r\n\r\n- Docker Support Forums - https://forums.docker.com\r\n- IRC - irc.freenode.net #docker channel\r\n- Post a question on StackOverflow, using the Docker tag\r\n\r\n---------------------------------------------------\r\nBUG REPORT INFORMATION\r\n---------------------------------------------------\r\nUse the commands below to provide key information from your environment:\r\nYou do NOT have to include this information if this is a FEATURE REQUEST\r\n-->\r\n\r\n**Description**\r\n\r\n<!--\r\nBriefly describe the problem you are having in a few paragraphs.\r\n-->\r\n\r\n**Steps to reproduce the issue:**\r\n1. Enable `--userns-remap=default` feature\r\n2. Buil/run a container with volumes added from the host (home folder) to the container\r\n\r\n**Describe the results you received:**\r\nAs discussed in [this issue](https://github.com/docker/docker/issues/25847#issuecomment-259111812) containers won't run with volumes mounted and `--userns-remap=default` enabled, a \"permission denied\" error is returned. I've also done `chmod 777` to the folders in question on the host with no luck.\r\n\r\nThis is the output after the build:\r\n```\r\n...\r\nCreating wwwdata\r\nCreating memcached\r\nCreating mariadb\r\n\r\nERROR: for mariadb  Cannot start service mariadb: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/cf4d6d7dbdeba48a656b5fe00fe2e7372cc9a8b766a0df0babc55d3f733872a1/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/etc/secret\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret: permission denied\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\"\r\n\r\nERROR: for wwwdata  Cannot start service wwwdata: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/www\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/eb556392efbac20549c4ecd95142767264d29f9603e71fc605f92d71528f56d3/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/www\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/www: permission denied\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\"\r\nERROR: Encountered errors while bringing up the project.\r\nMakefile:41: recipe for target 'up_prod' failed\r\nmake: *** [up_prod] Error 1\r\n\r\n```\r\nHere are `journalctl` logs:\r\n```\r\n...\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.029674535+02:00\" level=info msg=\"IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]\"\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.159778607+02:00\" level=error msg=\"containerd: start container\" error=\"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/www\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/eb556392efbac20549c4ecd95142767264d29f9603e71fc605f92d71528f56d3/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/www\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/www: permission denied\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\" id=67900b6e6f2ac06e17d9e6ff26e7a12884347960faba9b341f57364b35c3213d\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.162964900+02:00\" level=error msg=\"Create container failed with error: invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/www\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/eb556392efbac20549c4ecd95142767264d29f9603e71fc605f92d71528f56d3/rootfs\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/www\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/www: permission denied\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.421182256+02:00\" level=info msg=\"No non-localhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4]\"\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.421210327+02:00\" level=info msg=\"IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]\"\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.516917600+02:00\" level=error msg=\"Handler for POST /v1.21/containers/67900b6e6f2ac06e17d9e6ff26e7a12884347960faba9b341f57364b35c3213d/start returned error: invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/www\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/eb556392efbac20549c4ecd95142767264d29f9603e71fc605f92d71528f56d3/rootfs\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/www\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/www: permission denied\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.768576556+02:00\" level=info msg=\"No non-localhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4]\"\r\nNov 26 22:37:29 sam dockerd[7220]: time=\"2016-11-26T22:37:29.768610674+02:00\" level=info msg=\"IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]\"\r\nNov 26 22:37:30 sam dockerd[7220]: time=\"2016-11-26T22:37:29.907376608+02:00\" level=error msg=\"containerd: start container\" error=\"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/cf4d6d7dbdeba48a656b5fe00fe2e7372cc9a8b766a0df0babc55d3f733872a1/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/etc/secret\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret: permission denied\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\" id=ac153d20b8cd14705f95f00ac455927b2b60b5dca54624608dcccbcac5c6c12f\r\nNov 26 22:37:30 sam dockerd[7220]: time=\"2016-11-26T22:37:29.911870051+02:00\" level=error msg=\"Create container failed with error: invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/cf4d6d7dbdeba48a656b5fe00fe2e7372cc9a8b766a0df0babc55d3f733872a1/rootfs\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/etc/secret\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret: permission denied\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"\r\nNov 26 22:37:30 sam dockerd[7220]: time=\"2016-11-26T22:37:30.160272067+02:00\" level=error msg=\"Handler for POST /v1.21/containers/ac153d20b8cd14705f95f00ac455927b2b60b5dca54624608dcccbcac5c6c12f/start returned error: invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/231072.231072/devicemapper/mnt/cf4d6d7dbdeba48a656b5fe00fe2e7372cc9a8b766a0df0babc55d3f733872a1/rootfs\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/etc/secret\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"stat /home/achep/Apps/docker-lemp-stack/data/config/mariadb/secret: permission denied\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"\r\n```\r\n**Describe the results you expected:**\r\nObviously the thing should start and mount the shares.\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\nI've tried this on an Ubuntu 16.04 cloud instance which has Docker running on some externally mounted block devices as well as on my Ubuntu 16.04 laptop with basic, vanilla setup. Both boxes are running up-to-date or mainline kernels.\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   6b644ec\r\n Built:        Wed Oct 26 22:01:48 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   6b644ec\r\n Built:        Wed Oct 26 22:01:48 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 3\r\n Running: 1\r\n Paused: 0\r\n Stopped: 2\r\nImages: 28\r\nServer Version: 1.12.3\r\nStorage Driver: devicemapper\r\n Pool Name: docker-8:7-2228435-pool\r\n Pool Blocksize: 65.54 kB\r\n Base Device Size: 10.74 GB\r\n Backing Filesystem: ext4\r\n Data file: /dev/loop0\r\n Metadata file: /dev/loop1\r\n Data Space Used: 3.493 GB\r\n Data Space Total: 107.4 GB\r\n Data Space Available: 21.07 GB\r\n Metadata Space Used: 4.44 MB\r\n Metadata Space Total: 2.147 GB\r\n Metadata Space Available: 2.143 GB\r\n Thin Pool Minimum Free Space: 10.74 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Data loop file: /var/lib/docker/231072.231072/devicemapper/devicemapper/data\r\n WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.\r\n Metadata loop file: /var/lib/docker/231072.231072/devicemapper/devicemapper/metadata\r\n Library Version: 1.02.110 (2015-10-30)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null host bridge overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: apparmor seccomp\r\nKernel Version: 4.8.8-ph+\r\nOperating System: Ubuntu 16.04.1 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 4\r\nTotal Memory: 15.61 GiB\r\nName: sam\r\nID: PAD2:WRL5:LOZN:DIGV:IBPR:ZXP3:7VOF:SGZ5:SI42:2TMW:QIEV:V67R\r\nDocker Root Dir: /var/lib/docker/231072.231072\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nNot yet."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/263973213","html_url":"https://github.com/docker/docker/issues/28859#issuecomment-263973213","issue_url":"https://api.github.com/repos/docker/docker/issues/28859","id":263973213,"user":{"login":"shadeslayer","id":188976,"avatar_url":"https://avatars.githubusercontent.com/u/188976?v=3","gravatar_id":"","url":"https://api.github.com/users/shadeslayer","html_url":"https://github.com/shadeslayer","followers_url":"https://api.github.com/users/shadeslayer/followers","following_url":"https://api.github.com/users/shadeslayer/following{/other_user}","gists_url":"https://api.github.com/users/shadeslayer/gists{/gist_id}","starred_url":"https://api.github.com/users/shadeslayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shadeslayer/subscriptions","organizations_url":"https://api.github.com/users/shadeslayer/orgs","repos_url":"https://api.github.com/users/shadeslayer/repos","events_url":"https://api.github.com/users/shadeslayer/events{/privacy}","received_events_url":"https://api.github.com/users/shadeslayer/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T19:40:24Z","updated_at":"2016-11-30T19:40:24Z","body":"This looks like a kernel regression on our deployment. Some of our servers have varying kernel's and 3.13.0-52-generic on ubuntu 14.04 seems to be working just fine."}}
{"actor":{"display_login":"shadeslayer"}}